JUnit version 4.13.2
.20:03:47.670 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1779134627661
20:03:47.727 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
20:03:47.796 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
20:03:47.809 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskwwkcsW/test-tmp
20:03:47.816 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
20:03:47.819 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskwwkcsW/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
20:03:48.046 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

20:03:48.254 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
20:03:48.257 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

20:03:48.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.262257  6201 mini_oidc.cc:150] Starting JWKS server
20:03:48.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.266709  6201 webserver.cc:533] Webserver started at http://127.0.0.1:34643/ using document root <none> and password file <none>
20:03:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.267342  6201 mini_oidc.cc:186] Starting OIDC Discovery server
20:03:48.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.270182  6201 webserver.cc:533] Webserver started at http://127.0.0.1:42389/ using document root <none> and password file <none>
20:03:48.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T20:03:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
20:03:48.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T20:03:48Z Disabled control of system clock
20:03:48.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.336582  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:48.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:48.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/wal
20:03:48.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data
20:03:48.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:03:48.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:03:48.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:03:48.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:48.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:48.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:48.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/logs
20:03:48.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data/info.pb
20:03:48.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:48.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:03:48.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:03:48.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:03:48.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:03:48.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:48.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:48.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:48.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:03:48.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:03:48.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:03:48.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:03:48.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.126:43963
20:03:48.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.126
20:03:48.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:48.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:48.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:48.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:48.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:48.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:03:48.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:48.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:03:48.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34643/jwks.json with env {}
20:03:48.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.677399  6213 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:03:48.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.678133  6213 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:03:48.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.678604  6213 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:03:48.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.714851  6213 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
20:03:48.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.715178  6213 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:03:48.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.715377  6213 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
20:03:48.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.715721  6213 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
20:03:48.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.716022  6213 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
20:03:48.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.716189  6213 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34643/jwks.json
20:03:48.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.756722  6213 master_runner.cc:387] Master server non-default flags:
20:03:48.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data
20:03:48.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/wal
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.126:43963
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:03:48.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:03:48.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:03:48.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34643/jwks.json
20:03:48.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:48.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:48.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data/info.pb
20:03:48.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.126
20:03:48.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:48.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:03:48.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:48.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:03:48.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:03:48.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:48.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/logs
20:03:48.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:48.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:03:48.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:03:48.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:03:48.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:03:48.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:03:48.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:03:48.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:03:48.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:03:48.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:03:48.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.758550  6213 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:03:48.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:48.761075  6213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:03:48.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.775178  6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:50.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:48.777707  6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:50.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.008002  6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:50.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.010773  6221 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1228 milliseconds
20:03:50.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.010880  6213 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:03:50.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.012666  6213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:03:50.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.016116  6213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:03:50.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.017510  6213 hybrid_clock.cc:648] HybridClock initialized: now 1779134630017494 us; error 66 us; skew 500 ppm
20:03:50.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.018399  6213 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:03:50.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.025688  6213 webserver.cc:533] Webserver started at http://127.6.14.126:34407/ using document root <none> and password file <none>
20:03:50.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.026837  6213 fs_manager.cc:362] Metadata directory not provided
20:03:50.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.027043  6213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:03:50.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.027484  6213 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
20:03:50.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.032727  6213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data/instance:
20:03:50.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "93895b0070f441179c2a57863e02bd98"
20:03:50.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:50 on dist-test-slave-vt9x"
20:03:50.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.034075  6213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/wal/instance:
20:03:50.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "93895b0070f441179c2a57863e02bd98"
20:03:50.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:50 on dist-test-slave-vt9x"
20:03:50.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.042992  6213 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.009s	sys 0.001s
20:03:50.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.049811  6229 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:03:50.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.051307  6213 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.006s	sys 0.000s
20:03:50.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.051797  6213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/wal
20:03:50.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "93895b0070f441179c2a57863e02bd98"
20:03:50.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:50 on dist-test-slave-vt9x"
20:03:50.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.052290  6213 fs_report.cc:389] FS layout report
20:03:50.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:03:50.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/wal
20:03:50.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/wal
20:03:50.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data/data
20:03:50.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:03:50.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:03:50.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:03:50.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:03:50.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:03:50.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:03:50.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:03:50.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:03:50.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:03:50.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:03:50.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:03:50.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.111545  6213 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:03:50.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.113363  6213 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:03:50.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.113811  6213 kserver.cc:163] Server-wide thread pool size limit: 3276
20:03:50.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.200237  6213 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.126:43963
20:03:50.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.200289  6280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.126:43963 every 8 connection(s)
20:03:50.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.203519  6213 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/data/info.pb
20:03:50.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.209913  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 6213
20:03:50.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.210481  6201 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-0/wal/instance
20:03:50.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.210924  6281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:03:50.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.216200  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:50.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/wal
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:50.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/logs
20:03:50.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data/info.pb
20:03:50.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:50.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:03:50.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:03:50.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:03:50.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:03:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:03:50.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:03:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:03:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:03:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.125:41525
20:03:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.125
20:03:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:50.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:50.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:50.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:50.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:03:50.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:50.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:03:50.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34643/jwks.json with env {}
20:03:50.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.227547  6281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
20:03:50.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.261130  6281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
20:03:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.263984  6231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:50.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.269102  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:50.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.318724  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 1
20:03:50.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.323937  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:50.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.392377  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 2
20:03:50.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.398381  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:50.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.489850  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 3
20:03:50.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.494951  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:50.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.567574  6283 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:03:50.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.569365  6283 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:03:50.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.570194  6283 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:03:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.609416  6283 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
20:03:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.609745  6283 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:03:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.609942  6283 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
20:03:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.610131  6283 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
20:03:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.610316  6283 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
20:03:50.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.610462  6283 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34643/jwks.json
20:03:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.638425  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 4
20:03:50.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.643052  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:50.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.652930  6283 master_runner.cc:387] Master server non-default flags:
20:03:50.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/wal
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:03:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.125:41525
20:03:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:03:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:03:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:03:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34643/jwks.json
20:03:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data/info.pb
20:03:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.125
20:03:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:03:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:03:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:03:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/logs
20:03:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:03:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:03:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:03:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:03:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:03:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:03:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:03:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:03:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:03:50.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.654731  6283 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:03:50.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.657267  6283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:03:50.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.671020  6292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:50.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:50.942538  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 5
20:03:50.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.953735  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:51.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:51.501331  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 6
20:03:51.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:51.533078  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:52.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.075608  6291 debug-util.cc:398] Leaking SignalData structure 0x7b08000271e0 after lost signal to thread 6283
20:03:52.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.506701  6283 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.835s	user 0.682s	sys 1.147s
20:03:52.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.507084  6283 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.836s	user 0.682s	sys 1.147s
20:03:52.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:50.672372  6293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:52.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.511883  6295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:52.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.515707  6294 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1837 milliseconds
20:03:52.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.515748  6283 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:03:52.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.517589  6283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:03:52.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.521391  6283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:03:52.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.523289  6283 hybrid_clock.cc:648] HybridClock initialized: now 1779134632523210 us; error 102 us; skew 500 ppm
20:03:52.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.524320  6283 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:03:52.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.531198  6283 webserver.cc:533] Webserver started at http://127.6.14.125:41167/ using document root <none> and password file <none>
20:03:52.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.532279  6283 fs_manager.cc:362] Metadata directory not provided
20:03:52.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.532485  6283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:03:52.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.533140  6283 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
20:03:52.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.538242  6283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data/instance:
20:03:52.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0192a8161f894b7a98579c35df26a769"
20:03:52.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:52 on dist-test-slave-vt9x"
20:03:52.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.539650  6283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/wal/instance:
20:03:52.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0192a8161f894b7a98579c35df26a769"
20:03:52.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:52 on dist-test-slave-vt9x"
20:03:52.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.548118  6283 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.002s	sys 0.006s
20:03:52.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.554508  6304 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:03:52.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.555864  6283 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.007s	sys 0.000s
20:03:52.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.556205  6283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/wal
20:03:52.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0192a8161f894b7a98579c35df26a769"
20:03:52.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:52 on dist-test-slave-vt9x"
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.556543  6283 fs_report.cc:389] FS layout report
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/wal
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/wal
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data/data
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:03:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:03:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:03:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:03:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:03:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:03:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:03:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:03:52.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.593691  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 7
20:03:52.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.601578  6281 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.125:41525: Network error: Client connection negotiation failed: client connection to 127.6.14.125:41525: connect: Connection refused (error 111)
20:03:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.633742  6283 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:03:52.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.635839  6283 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:03:52.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.636338  6283 kserver.cc:163] Server-wide thread pool size limit: 3276
20:03:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.724478  6283 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.125:41525
20:03:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.724566  6357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.125:41525 every 8 connection(s)
20:03:52.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.728085  6283 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/data/info.pb
20:03:52.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.731132  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 6283
20:03:52.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.731702  6201 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-1/wal/instance
20:03:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.735029  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/wal
20:03:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/logs
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data/info.pb
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:03:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:03:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:03:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.124:44305
20:03:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.124
20:03:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:52.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:52.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:52.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:03:52.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:52.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:03:52.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34643/jwks.json with env {}
20:03:52.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.740253  6358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:03:52.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.761085  6358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
20:03:52.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.790201  6358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
20:03:52.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.806905  6358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
20:03:52.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.809957  6307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:52.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.814512  6358 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.124:44305: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:52.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.864084  6358 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } attempt: 1
20:03:52.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.868892  6358 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.124:44305: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:52.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:52.937470  6358 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } attempt: 2
20:03:52.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:52.942194  6358 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.124:44305: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:53.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:53.033746  6358 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } attempt: 3
20:03:53.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.038326  6358 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.124:44305: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:53.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.105037  6360 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:03:53.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.105683  6360 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:03:53.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.106139  6360 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:03:53.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.146816  6360 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
20:03:53.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.147378  6360 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:03:53.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.147774  6360 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
20:03:53.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.148051  6360 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
20:03:53.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.148358  6360 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
20:03:53.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.148850  6360 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34643/jwks.json
20:03:53.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:53.181806  6358 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } attempt: 4
20:03:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.186942  6358 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.124:44305: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:53.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:53.207855  6360 master_runner.cc:387] Master server non-default flags:
20:03:53.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/wal
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:03:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.124:44305
20:03:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:03:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:03:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:03:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34643/jwks.json
20:03:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data/info.pb
20:03:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.124
20:03:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/logs
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:03:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:03:53.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:03:53.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:03:53.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:03:53.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:03:53.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:03:53.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:03:53.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:03:53.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:53.210040  6360 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:03:53.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:53.212822  6360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:03:53.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.225562  6370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:53.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:53.486498  6358 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } attempt: 5
20:03:53.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.494144  6358 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.124:44305: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:54.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.041805  6358 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } attempt: 6
20:03:54.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.055272  6358 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.14.124:44305: Network error: Client connection negotiation failed: client connection to 127.6.14.124:44305: connect: Connection refused (error 111)
20:03:54.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:53.227566  6371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:54.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.487012  6373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:54.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.489766  6372 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1257 milliseconds
20:03:54.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.489864  6360 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:03:54.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.491324  6360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:03:54.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.494495  6360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:03:54.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.495896  6360 hybrid_clock.cc:648] HybridClock initialized: now 1779134634495861 us; error 43 us; skew 500 ppm
20:03:54.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.496876  6360 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:03:54.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.504078  6360 webserver.cc:533] Webserver started at http://127.6.14.124:37223/ using document root <none> and password file <none>
20:03:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.505239  6360 fs_manager.cc:362] Metadata directory not provided
20:03:54.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.505508  6360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:03:54.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.506124  6360 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
20:03:54.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.511258  6360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data/instance:
20:03:54.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4734867fcd4d4571b9d253d82e616d32"
20:03:54.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:54 on dist-test-slave-vt9x"
20:03:54.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.512677  6360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/wal/instance:
20:03:54.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4734867fcd4d4571b9d253d82e616d32"
20:03:54.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:54 on dist-test-slave-vt9x"
20:03:54.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.521047  6360 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.002s	sys 0.005s
20:03:54.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.527593  6382 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:03:54.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.528872  6360 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
20:03:54.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.529286  6360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/wal
20:03:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4734867fcd4d4571b9d253d82e616d32"
20:03:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:54 on dist-test-slave-vt9x"
20:03:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.529734  6360 fs_report.cc:389] FS layout report
20:03:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:03:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/wal
20:03:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/wal
20:03:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data/data
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:03:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:03:54.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:03:54.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:03:54.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:03:54.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.597707  6360 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:03:54.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.600039  6360 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:03:54.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.600615  6360 kserver.cc:163] Server-wide thread pool size limit: 3276
20:03:54.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.688766  6360 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.124:44305
20:03:54.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.688836  6433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.124:44305 every 8 connection(s)
20:03:54.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.692121  6281 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } attempt: 8
20:03:54.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.692432  6360 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/data/info.pb
20:03:54.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.695401  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 6360
20:03:54.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.695959  6201 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/master-2/wal/instance
20:03:54.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.708269  6435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:03:54.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.716492  6281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
20:03:54.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.739609  6435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } has no permanent_uuid. Determining permanent_uuid...
20:03:54.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.771356  6435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
20:03:54.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.771703  6281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: Bootstrap starting.
20:03:54.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.780344  6281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: Neither blocks nor log segments found. Creating new log.
20:03:54.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.783865  6281 log.cc:826] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: Log is configured to *not* fsync() on all Append() calls
20:03:54.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.786502  6435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
20:03:54.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.791927  6281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: No bootstrap required, opened a new log
20:03:54.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.805369  6435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32: Bootstrap starting.
20:03:54.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.813494  6435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32: Neither blocks nor log segments found. Creating new log.
20:03:54.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.815639  6435 log.cc:826] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32: Log is configured to *not* fsync() on all Append() calls
20:03:54.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.817379  6281 raft_consensus.cc:359] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:54.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.818569  6281 raft_consensus.cc:385] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:03:54.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.818892  6281 raft_consensus.cc:740] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93895b0070f441179c2a57863e02bd98, State: Initialized, Role: FOLLOWER
20:03:54.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.819895  6281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:54.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.822373  6435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32: No bootstrap required, opened a new log
20:03:54.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.826532  6445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:54.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.827342  6281 sys_catalog.cc:565] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [sys.catalog]: configured and running, proceeding with master startup.
20:03:54.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.827419  6445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [sys.catalog]: This master's current role is: FOLLOWER
20:03:54.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.845597  6435 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:54.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.846449  6435 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:03:54.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.846735  6435 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4734867fcd4d4571b9d253d82e616d32, State: Initialized, Role: FOLLOWER
20:03:54.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.847934  6435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:54.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.855967  6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:54.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.857517  6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: This master's current role is: FOLLOWER
20:03:54.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.858042  6435 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: configured and running, proceeding with master startup.
20:03:54.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.867894  6457 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:03:54.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.868925  6457 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:03:54.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.877604  6235 tablet.cc:2404] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:03:54.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.868134  6213 master_runner.cc:187] Error getting master registration for 127.6.14.125:41525: OK, instance_id { permanent_uuid: "0192a8161f894b7a98579c35df26a769" instance_seqno: 1779134632700622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:03:54.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.912524  6213 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
20:03:54.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.917618  6469 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:03:54.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:54.918850  6469 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:03:54.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.914932  6360 master_runner.cc:187] Error getting master registration for 127.6.14.125:41525: OK, instance_id { permanent_uuid: "0192a8161f894b7a98579c35df26a769" instance_seqno: 1779134632700622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:03:54.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:54.955354  6360 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
20:03:55.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.038496  6454 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:03:55.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.039191  6454 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:55.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.042270  6454 leader_election.cc:290] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93895b0070f441179c2a57863e02bd98 (127.6.14.126:43963), 0192a8161f894b7a98579c35df26a769 (127.6.14.125:41525)
20:03:55.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.043713  6256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4734867fcd4d4571b9d253d82e616d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93895b0070f441179c2a57863e02bd98" is_pre_election: true
20:03:55.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.044811  6256 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4734867fcd4d4571b9d253d82e616d32 in term 0.
20:03:55.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.046913  6384 leader_election.cc:304] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4734867fcd4d4571b9d253d82e616d32, 93895b0070f441179c2a57863e02bd98; no voters: 
20:03:55.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.048241  6454 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Leader pre-election won for term 1
20:03:55.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.048875  6454 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:03:55.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.050275  6454 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 0 FOLLOWER]: Advancing to term 1
20:03:55.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.060498  6454 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:55.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.064014  6454 leader_election.cc:290] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [CANDIDATE]: Term 1 election: Requested vote from peers 93895b0070f441179c2a57863e02bd98 (127.6.14.126:43963), 0192a8161f894b7a98579c35df26a769 (127.6.14.125:41525)
20:03:55.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.066170  6256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4734867fcd4d4571b9d253d82e616d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93895b0070f441179c2a57863e02bd98"
20:03:55.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.066855  6256 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [term 0 FOLLOWER]: Advancing to term 1
20:03:55.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.076618  6256 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4734867fcd4d4571b9d253d82e616d32 in term 1.
20:03:55.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.078684  6384 leader_election.cc:304] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4734867fcd4d4571b9d253d82e616d32, 93895b0070f441179c2a57863e02bd98; no voters: 
20:03:55.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.080883  6454 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 1 FOLLOWER]: Leader election won for term 1
20:03:55.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.083217  6454 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [term 1 LEADER]: Becoming Leader. State: Replica: 4734867fcd4d4571b9d253d82e616d32, State: Running, Role: LEADER
20:03:55.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.084712  6454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:55.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:55.107753  6388 tablet.cc:2404] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:03:55.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.106285  6471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4734867fcd4d4571b9d253d82e616d32. Latest consensus state: current_term: 1 leader_uuid: "4734867fcd4d4571b9d253d82e616d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.114837  6471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: This master's current role is: LEADER
20:03:55.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.116211  6358 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } attempt: 7
20:03:55.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.133226  6474 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
20:03:55.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.142637  6474 catalog_manager.cc:1519] Initializing Kudu cluster ID...
20:03:55.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.241020  6358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769: Bootstrap starting.
20:03:55.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.254186  6256 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [term 1 FOLLOWER]: Refusing update from remote peer 4734867fcd4d4571b9d253d82e616d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:03:55.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.257014  6454 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
20:03:55.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.320556  6445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4734867fcd4d4571b9d253d82e616d32. Latest consensus state: current_term: 1 leader_uuid: "4734867fcd4d4571b9d253d82e616d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.321396  6445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [sys.catalog]: This master's current role is: FOLLOWER
20:03:55.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.320508  6358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769: Neither blocks nor log segments found. Creating new log.
20:03:55.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.327903  6475 mvcc.cc:204] Tried to move back new op lower bound from 7287335465950846976 to 7287335465318674432. Current Snapshot: MvccSnapshot[applied={T|T < 7287335465950846976}]
20:03:55.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.335093  6358 log.cc:826] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769: Log is configured to *not* fsync() on all Append() calls
20:03:55.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.335045  6471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4734867fcd4d4571b9d253d82e616d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.335934  6471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: This master's current role is: LEADER
20:03:55.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.338400  6474 catalog_manager.cc:1382] Generated new cluster ID: 6ac4802fd4ba463bb998be8720237ffd
20:03:55.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.338796  6474 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
20:03:55.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.347523  6445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4734867fcd4d4571b9d253d82e616d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.348378  6445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98 [sys.catalog]: This master's current role is: FOLLOWER
20:03:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.365427  6358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769: No bootstrap required, opened a new log
20:03:55.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.419842  6474 catalog_manager.cc:1405] Generated new certificate authority record
20:03:55.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.422358  6474 catalog_manager.cc:1539] Loading token signing keys...
20:03:55.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.449355  6358 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:55.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.453269  6358 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:03:55.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.454770  6358 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0192a8161f894b7a98579c35df26a769, State: Initialized, Role: FOLLOWER
20:03:55.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.464097  6474 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32: Generated new TSK 0
20:03:55.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.458278  6358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } }
20:03:55.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.465536  6474 catalog_manager.cc:1549] Initializing in-progress tserver states...
20:03:55.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.471550  6481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.472469  6481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [sys.catalog]: This master's current role is: FOLLOWER
20:03:55.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.474929  6358 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [sys.catalog]: configured and running, proceeding with master startup.
20:03:55.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.475605  6331 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [term 0 FOLLOWER]: Advancing to term 1
20:03:55.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.481173  6332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4734867fcd4d4571b9d253d82e616d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0192a8161f894b7a98579c35df26a769"
20:03:55.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.479974  6333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4734867fcd4d4571b9d253d82e616d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0192a8161f894b7a98579c35df26a769" is_pre_election: true
20:03:55.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.494596  6331 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [term 1 FOLLOWER]: Refusing update from remote peer 4734867fcd4d4571b9d253d82e616d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.505076  6471 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.506827  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:03:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:55.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:55.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:03:55.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:03:55.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:0
20:03:55.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:03:55.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:03:55.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:55.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:55.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:55.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:55.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:55.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:03:55.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.593094  6481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4734867fcd4d4571b9d253d82e616d32. Latest consensus state: current_term: 1 leader_uuid: "4734867fcd4d4571b9d253d82e616d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.594007  6481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [sys.catalog]: This master's current role is: FOLLOWER
20:03:55.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.593223  6489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4734867fcd4d4571b9d253d82e616d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.595294  6489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769 [sys.catalog]: This master's current role is: FOLLOWER
20:03:55.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.633672  6499 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
20:03:55.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.875725  6457 catalog_manager.cc:1294] Loaded cluster ID: 6ac4802fd4ba463bb998be8720237ffd
20:03:55.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.876652  6457 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: loading cluster ID for follower catalog manager: success
20:03:55.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.885689  6457 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: acquiring CA information for follower catalog manager: success
20:03:55.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.892102  6457 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 93895b0070f441179c2a57863e02bd98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:03:55.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.975566  6471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4734867fcd4d4571b9d253d82e616d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93895b0070f441179c2a57863e02bd98" member_type: VOTER last_known_addr { host: "127.6.14.126" port: 43963 } } peers { permanent_uuid: "0192a8161f894b7a98579c35df26a769" member_type: VOTER last_known_addr { host: "127.6.14.125" port: 41525 } } peers { permanent_uuid: "4734867fcd4d4571b9d253d82e616d32" member_type: VOTER last_known_addr { host: "127.6.14.124" port: 44305 } } }
20:03:55.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:55.977216  6471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4734867fcd4d4571b9d253d82e616d32 [sys.catalog]: This master's current role is: LEADER
20:03:56.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:56.110234  6484 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:03:56.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:56.110926  6484 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:03:56.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:56.111747  6484 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:03:56.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:56.177762  6484 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:03:56.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:56.179234  6484 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.65
20:03:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:56.251628  6484 tablet_server_runner.cc:78] Tablet server non-default flags:
20:03:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:56.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:0
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:56.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:03:56.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:03:56.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:56.256832  6484 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:03:56.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:56.259881  6484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:03:56.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:56.287997  6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:56.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:56.646530  6499 catalog_manager.cc:1294] Loaded cluster ID: 6ac4802fd4ba463bb998be8720237ffd
20:03:56.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:56.646869  6499 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769: loading cluster ID for follower catalog manager: success
20:03:56.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:56.672327  6499 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769: acquiring CA information for follower catalog manager: success
20:03:56.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:56.686700  6499 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 0192a8161f894b7a98579c35df26a769: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:03:58.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:57.694600  6506 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 6484
20:03:58.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:58.824774  6484 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.531s	user 0.000s	sys 0.002s
20:03:58.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:58.825927  6484 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.532s	user 0.000s	sys 0.002s
20:03:58.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:58.823945  6509 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2529 milliseconds
20:03:58.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:56.292567  6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:58.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.828657  6484 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:03:58.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:58.831245  6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:03:58.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.833354  6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:03:58.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.836887  6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:03:58.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.838588  6484 hybrid_clock.cc:648] HybridClock initialized: now 1779134638838542 us; error 49 us; skew 500 ppm
20:03:58.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.839795  6484 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:03:58.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.849035  6484 webserver.cc:533] Webserver started at http://127.6.14.65:44657/ using document root <none> and password file <none>
20:03:58.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.850466  6484 fs_manager.cc:362] Metadata directory not provided
20:03:58.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.850742  6484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:03:58.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.851320  6484 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
20:03:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.858731  6484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/instance:
20:03:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "993f936550594542941a497816c4b19d"
20:03:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:58 on dist-test-slave-vt9x"
20:03:58.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.860477  6484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal/instance:
20:03:58.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "993f936550594542941a497816c4b19d"
20:03:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:58 on dist-test-slave-vt9x"
20:03:58.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.871982  6484 fs_manager.cc:696] Time spent creating directory manager: real 0.011s	user 0.010s	sys 0.001s
20:03:58.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.880760  6520 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:03:58.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.882395  6484 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.000s
20:03:58.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.882812  6484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:03:58.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "993f936550594542941a497816c4b19d"
20:03:58.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:58 on dist-test-slave-vt9x"
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.883591  6484 fs_report.cc:389] FS layout report
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/data
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:03:58.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:03:58.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:03:58.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:03:58.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:03:58.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:03:58.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:03:58.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:03:58.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.952329  6484 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:03:58.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.954639  6484 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:03:58.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.955255  6484 kserver.cc:163] Server-wide thread pool size limit: 3276
20:03:58.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.959251  6484 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:03:58.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.965822  6484 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
20:03:58.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.966121  6484 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
20:03:58.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.966460  6484 ts_tablet_manager.cc:616] Registered 0 tablets
20:03:58.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:58.966706  6484 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
20:03:59.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.155164  6484 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.65:40343
20:03:59.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.155256  6634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.65:40343 every 8 connection(s)
20:03:59.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.159529  6484 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:03:59.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.160540  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 6484
20:03:59.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.161242  6201 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal/instance
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.181315  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:03:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:0
20:03:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:03:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:03:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:59.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:03:59.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.238700  6635 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:03:59.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.239383  6635 heartbeater.cc:461] Registering TS with master...
20:03:59.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.241026  6635 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:03:59.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.241178  6637 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:03:59.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.241598  6637 heartbeater.cc:461] Registering TS with master...
20:03:59.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.242676  6637 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:03:59.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.245795  6399 ts_manager.cc:194] Registered new tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:03:59.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.246325  6246 ts_manager.cc:194] Registered new tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:03:59.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.253067  6636 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:03:59.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.253496  6636 heartbeater.cc:461] Registering TS with master...
20:03:59.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.253067  6399 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.65:40311
20:03:59.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.254546  6636 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:03:59.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.258371  6323 ts_manager.cc:194] Registered new tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:03:59.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:59.616133  6645 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:03:59.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:59.616837  6645 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:03:59.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:59.617553  6645 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:03:59.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:59.656399  6645 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:03:59.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:59.657562  6645 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.66
20:03:59.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.714149  6645 tablet_server_runner.cc:78] Tablet server non-default flags:
20:03:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:03:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:03:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:03:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:03:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:03:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:03:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:0
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:03:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:03:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:03:59.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.719586  6645 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:03:59.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:03:59.723579  6645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:03:59.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:59.743690  6651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:00.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:00.264416  6635 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:01.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:01.147927  6650 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 6645
20:04:01.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:01.189177  6645 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.444s	user 0.552s	sys 0.882s
20:04:01.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:03:59.746381  6652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:01.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:01.189558  6645 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.445s	user 0.552s	sys 0.882s
20:04:01.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.198849  6645 server_base.cc:1061] running on GCE node
20:04:01.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:01.198848  6656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:01.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.200551  6645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:01.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.203248  6645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:01.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.204694  6645 hybrid_clock.cc:648] HybridClock initialized: now 1779134641204616 us; error 77 us; skew 500 ppm
20:04:01.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.205776  6645 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.213552  6645 webserver.cc:533] Webserver started at http://127.6.14.66:41631/ using document root <none> and password file <none>
20:04:01.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.214740  6645 fs_manager.cc:362] Metadata directory not provided
20:04:01.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.215024  6645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:01.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.215652  6645 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
20:04:01.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.221024  6645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/instance:
20:04:01.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:01.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:01 on dist-test-slave-vt9x"
20:04:01.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.222417  6645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal/instance:
20:04:01.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:01.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:01 on dist-test-slave-vt9x"
20:04:01.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.231261  6645 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.005s
20:04:01.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.238273  6662 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:01.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.239583  6645 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.002s
20:04:01.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.240059  6645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:01.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:01.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:01 on dist-test-slave-vt9x"
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.240550  6645 fs_report.cc:389] FS layout report
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/data
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:01.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:01.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:01.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:01.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.297595  6645 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:01.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.299536  6645 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:01.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.300212  6645 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:01.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.303583  6645 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:01.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.308934  6645 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
20:04:01.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.309191  6645 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
20:04:01.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.309551  6645 ts_tablet_manager.cc:616] Registered 0 tablets
20:04:01.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.309800  6645 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
20:04:01.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.501719  6645 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.66:36169
20:04:01.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.501839  6774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.66:36169 every 8 connection(s)
20:04:01.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.504963  6645 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:01.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.514487  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 6645
20:04:01.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.515120  6201 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal/instance
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.527582  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:0
20:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.557353  6775 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.557976  6775 heartbeater.cc:461] Registering TS with master...
20:04:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.559864  6775 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:01.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.563860  6399 ts_manager.cc:194] Registered new tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:01.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.565374  6399 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.66:55063
20:04:01.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.566183  6776 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:01.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.566614  6776 heartbeater.cc:461] Registering TS with master...
20:04:01.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.567720  6776 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:01.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.570878  6323 ts_manager.cc:194] Registered new tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:01.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.571524  6777 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:01.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.571856  6777 heartbeater.cc:461] Registering TS with master...
20:04:01.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.572541  6777 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:01.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:01.575201  6246 ts_manager.cc:194] Registered new tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:02.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:02.025009  6783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:02.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:02.025651  6783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:02.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:02.026074  6783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:02.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:02.062495  6783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:02.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:02.063732  6783 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.67
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:02.106566  6783 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:0
20:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:02.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:02.108873  6783 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:02.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:02.111083  6783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:02.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:02.126960  6791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:02.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:02.573060  6775 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:03.537416  6790 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 6783
20:04:03.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:03.762490  6783 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.635s	user 0.568s	sys 0.965s
20:04:03.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:02.127692  6792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:03.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:03.763211  6783 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.636s	user 0.568s	sys 0.965s
20:04:03.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:03.765317  6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:03.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:03.770637  6793 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1641 milliseconds
20:04:03.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.770645  6783 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:03.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.772229  6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:03.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.774905  6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:03.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.776314  6783 hybrid_clock.cc:648] HybridClock initialized: now 1779134643776257 us; error 69 us; skew 500 ppm
20:04:03.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.777441  6783 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:03.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.785552  6783 webserver.cc:533] Webserver started at http://127.6.14.67:45941/ using document root <none> and password file <none>
20:04:03.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.786612  6783 fs_manager.cc:362] Metadata directory not provided
20:04:03.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.786806  6783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:03.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.787214  6783 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
20:04:03.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.792060  6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/instance:
20:04:03.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:03.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:03 on dist-test-slave-vt9x"
20:04:03.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.793187  6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal/instance:
20:04:03.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:03.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:03 on dist-test-slave-vt9x"
20:04:03.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.801110  6783 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.002s	sys 0.006s
20:04:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.807113  6802 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.808334  6783 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
20:04:03.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.808727  6783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:03.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:03.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:03 on dist-test-slave-vt9x"
20:04:03.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.809192  6783 fs_report.cc:389] FS layout report
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/data
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:03.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.863157  6783 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:03.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.865069  6783 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:03.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.865665  6783 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:03.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.868904  6783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:03.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.874097  6783 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
20:04:03.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.874359  6783 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
20:04:03.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.874740  6783 ts_tablet_manager.cc:616] Registered 0 tablets
20:04:03.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:03.874943  6783 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
20:04:04.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.266853  6783 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.67:33469
20:04:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.267059  6914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.67:33469 every 8 connection(s)
20:04:04.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.271051  6783 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:04.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.277173  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 6783
20:04:04.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.277789  6201 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal/instance
20:04:04.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.356094  6915 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:04.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.356863  6915 heartbeater.cc:461] Registering TS with master...
20:04:04.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.357214  6916 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:04.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.357700  6916 heartbeater.cc:461] Registering TS with master...
20:04:04.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.359470  6915 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:04.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.359690  6916 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:04.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.364907  6399 ts_manager.cc:194] Registered new tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:04.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.368434  6399 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.67:43633
20:04:04.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.371080  6323 ts_manager.cc:194] Registered new tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:04.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.377694  6919 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:04.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.378211  6919 heartbeater.cc:461] Registering TS with master...
20:04:04.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.379369  6919 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:04.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.382616  6246 ts_manager.cc:194] Registered new tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:04.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:04.394181  6201 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
20:04:04.415 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
20:04:04.417 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

20:04:04.429 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.6.14.126"
      port: 43963
    }
    bound_http_address {
      host: "127.6.14.126"
      port: 34407
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.6.14.125"
      port: 41525
    }
    bound_http_address {
      host: "127.6.14.125"
      port: 41167
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.6.14.124"
      port: 44305
    }
    bound_http_address {
      host: "127.6.14.124"
      port: 37223
    }
  }
}

20:04:04.434 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

20:04:04.437 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.6.14.65"
      port: 40343
    }
    bound_http_address {
      host: "127.6.14.65"
      port: 44657
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.6.14.66"
      port: 36169
    }
    bound_http_address {
      host: "127.6.14.66"
      port: 41631
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.6.14.67"
      port: 33469
    }
    bound_http_address {
      host: "127.6.14.67"
      port: 45941
    }
  }
}

20:04:04.438 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
20:04:04.703 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
20:04:04.804 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
20:04:04.981 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.6.14.125:41525(127.6.14.125:41525)] connecting to peer
20:04:05.047 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.6.14.124:44305(127.6.14.124:44305)] connecting to peer
20:04:05.049 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.6.14.126:43963(127.6.14.126:43963)] connecting to peer
20:04:05.069 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.6.14.126:43963(127.6.14.126:43963)] Successfully connected to peer
20:04:05.071 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.6.14.124:44305(127.6.14.124:44305)] Successfully connected to peer
20:04:05.075 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.6.14.125:41525(127.6.14.125:41525)] Successfully connected to peer
20:04:05.195 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.6.14.125
20:04:05.195 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.6.14.124
20:04:05.195 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.6.14.126
20:04:05.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.375703  6915 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:05.456 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x169e434e, L:/127.0.0.1:36116 - R:/127.6.14.124:44305] using SASL/PLAIN
20:04:05.456 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xdaa52a50, L:/127.0.0.1:49432 - R:/127.6.14.125:41525] using SASL/PLAIN
20:04:05.458 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x76d6ce70, L:/127.0.0.1:51938 - R:/127.6.14.126:43963] using SASL/PLAIN
20:04:05.504 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
20:04:05.505 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=563), Traces: [0ms] refreshing cache from master, [24ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.14.125:41525, [100ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.14.124:44305, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.14.126:43963, [531ms] Sub RPC ConnectToMaster: received response from server master-127.6.14.126:43963: OK, [531ms] Sub RPC ConnectToMaster: received response from server master-127.6.14.124:44305: OK, [532ms] Sub RPC ConnectToMaster: received response from server master-127.6.14.125:41525: OK, deferred=null) after lookup
20:04:05.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.564103  6399 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928"
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
20:04:05.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
20:04:05.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
20:04:05.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
20:04:05.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
20:04:05.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
20:04:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
20:04:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
20:04:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
20:04:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
20:04:05.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:05.570997  6399 catalog_manager.cc:7058] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
20:04:05.698 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table dd90b95f74d64cc592ae912efc32a65c
20:04:05.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.758670  6850 tablet_service.cc:1511] Processing CreateTablet for tablet 598f9144972d45d6bb2d77bd7f869093 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928 [id=dd90b95f74d64cc592ae912efc32a65c]), partition=RANGE (key) PARTITION UNBOUNDED
20:04:05.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.773505  6850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598f9144972d45d6bb2d77bd7f869093. 1 dirs total, 0 dirs full, 0 dirs failed
20:04:05.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.813920  6568 tablet_service.cc:1511] Processing CreateTablet for tablet 598f9144972d45d6bb2d77bd7f869093 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928 [id=dd90b95f74d64cc592ae912efc32a65c]), partition=RANGE (key) PARTITION UNBOUNDED
20:04:05.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.822211  6568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598f9144972d45d6bb2d77bd7f869093. 1 dirs total, 0 dirs full, 0 dirs failed
20:04:05.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.827190  6710 tablet_service.cc:1511] Processing CreateTablet for tablet 598f9144972d45d6bb2d77bd7f869093 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928 [id=dd90b95f74d64cc592ae912efc32a65c]), partition=RANGE (key) PARTITION UNBOUNDED
20:04:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.831794  6710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598f9144972d45d6bb2d77bd7f869093. 1 dirs total, 0 dirs full, 0 dirs failed
20:04:05.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.907686  6952 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap starting.
20:04:05.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.912940  6951 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap starting.
20:04:05.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.916565  6953 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap starting.
20:04:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.926879  6952 tablet_bootstrap.cc:654] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Neither blocks nor log segments found. Creating new log.
20:04:05.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.932444  6953 tablet_bootstrap.cc:654] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Neither blocks nor log segments found. Creating new log.
20:04:05.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.933887  6952 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Log is configured to *not* fsync() on all Append() calls
20:04:05.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.934469  6951 tablet_bootstrap.cc:654] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Neither blocks nor log segments found. Creating new log.
20:04:05.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.937093  6953 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Log is configured to *not* fsync() on all Append() calls
20:04:05.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.941083  6951 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Log is configured to *not* fsync() on all Append() calls
20:04:05.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.951633  6952 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: No bootstrap required, opened a new log
20:04:05.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.951328  6953 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: No bootstrap required, opened a new log
20:04:05.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.953146  6953 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent bootstrapping tablet: real 0.038s	user 0.019s	sys 0.014s
20:04:05.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.953179  6952 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent bootstrapping tablet: real 0.053s	user 0.008s	sys 0.030s
20:04:05.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.976758  6951 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: No bootstrap required, opened a new log
20:04:05.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.978492  6951 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent bootstrapping tablet: real 0.073s	user 0.019s	sys 0.039s
20:04:06.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.998910  6953 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:05.999503  6952 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.000546  6953 raft_consensus.cc:385] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:04:06.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.000833  6952 raft_consensus.cc:385] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:04:06.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.000934  6953 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Initialized, Role: FOLLOWER
20:04:06.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.001261  6952 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993f936550594542941a497816c4b19d, State: Initialized, Role: FOLLOWER
20:04:06.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.002563  6953 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.003039  6952 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.016453  6952 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent starting tablet: real 0.062s	user 0.050s	sys 0.007s
20:04:06.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.017005  6953 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent starting tablet: real 0.064s	user 0.032s	sys 0.027s
20:04:06.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.041741  6951 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.043959  6951 raft_consensus.cc:385] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:04:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.048359  6951 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35e6d7e70184db4b58023c7bcd30482, State: Initialized, Role: FOLLOWER
20:04:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.054816  6951 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.072500  6951 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent starting tablet: real 0.089s	user 0.054s	sys 0.007s
20:04:06.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:06.083091  6922 tablet.cc:2404] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:04:06.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:06.209730  6642 tablet.cc:2404] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:04:06.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:06.291486  6778 tablet.cc:2404] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:04:06.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.293329  6958 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:04:06.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.294224  6958 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.298167  6958 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:06.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.302181  6957 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:04:06.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.302815  6957 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.317267  6957 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:06.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.347438  6870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a" is_pre_election: true
20:04:06.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.348515  6870 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 0.
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.351274  6521 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d; no voters: 
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.352391  6958 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Leader pre-election won for term 1
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.352777  6958 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.352761  6959 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.353085  6958 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 0 FOLLOWER]: Advancing to term 1
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.353433  6959 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.356266  6730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:06.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.357106  6730 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 0.
20:04:06.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.363065  6730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:06.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.364233  6730 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 0.
20:04:06.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.363524  6805 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482; no voters: 
20:04:06.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.365522  6957 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
20:04:06.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.366034  6957 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:04:06.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.366457  6957 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 0 FOLLOWER]: Advancing to term 1
20:04:06.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.367872  6959 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:06.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.364800  6958 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.374508  6590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d" is_pre_election: true
20:04:06.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.375828  6958 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 1 election: Requested vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:06.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.376173  6590 raft_consensus.cc:2393] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31b9b4da6f9945578c783b9a7a597b5a in current term 1: Already voted for candidate 993f936550594542941a497816c4b19d in this term.
20:04:06.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.376139  6870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:06.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.376350  6730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:06.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.377023  6730 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 0 FOLLOWER]: Advancing to term 1
20:04:06.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.377538  6957 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.379278  6870 raft_consensus.cc:2393] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 993f936550594542941a497816c4b19d in current term 1: Already voted for candidate 31b9b4da6f9945578c783b9a7a597b5a in this term.
20:04:06.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.380121  6957 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 1 election: Requested vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:06.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.380760  6729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:06.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.384094  6590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d"
20:04:06.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.385107  6590 raft_consensus.cc:2393] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31b9b4da6f9945578c783b9a7a597b5a in current term 1: Already voted for candidate 993f936550594542941a497816c4b19d in this term.
20:04:06.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.386288  6804 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a; no voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482
20:04:06.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.387171  6957 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
20:04:06.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.390558  6730 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 1.
20:04:06.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.392093  6523 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:06.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.393443  6958 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Leader election won for term 1
20:04:06.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.395999  6870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a" is_pre_election: true
20:04:06.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.396359  6958 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 LEADER]: Becoming Leader. State: Replica: 993f936550594542941a497816c4b19d, State: Running, Role: LEADER
20:04:06.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.397879  6870 raft_consensus.cc:2393] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e35e6d7e70184db4b58023c7bcd30482 in current term 1: Already voted for candidate 31b9b4da6f9945578c783b9a7a597b5a in this term.
20:04:06.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.399134  6958 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:06.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.403569  6590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d" is_pre_election: true
20:04:06.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.405682  6664 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d
20:04:06.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.409636  6959 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
20:04:06.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.428313  6398 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d reported cstate change: term changed from 0 to 1, leader changed from <none> to 993f936550594542941a497816c4b19d (127.6.14.65). New cstate: current_term: 1 leader_uuid: "993f936550594542941a497816c4b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: HEALTHY } } }
20:04:06.575 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table dd90b95f74d64cc592ae912efc32a65c
20:04:06.600 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:06.609 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299991, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:06.611 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=null) after lookup
20:04:06.623 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:06.626 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] Successfully connected to peer
20:04:06.749 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xec7428f4, L:/127.0.0.1:58520 - R:/127.6.14.65:40343] using TOKEN/null
20:04:06.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.778992  6730 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 1 FOLLOWER]: Refusing update from remote peer 993f936550594542941a497816c4b19d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:04:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.779851  6870 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Refusing update from remote peer 993f936550594542941a497816c4b19d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:04:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.780948  6958 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:04:06.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:06.784192  6972 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
20:04:07.048 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:07.050 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.65:40343
20:04:07.052 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

20:04:07.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:07.053092  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 6484
20:04:07.085 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:07.088 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] handling channelInactive
20:04:07.089 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state READY due to: connection closed
20:04:07.090 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: connection closed
20:04:07.109 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:07.109 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:07.110 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [3ms] received response from server 993f936550594542941a497816c4b19d: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@701958279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@5f5e4a7d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f4a124 -> wakeup thread Time-limited test)) after lookup
20:04:07.110 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:07.113 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.113 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.113 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.129 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:07.129 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:07.130 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [3ms] received response from server 993f936550594542941a497816c4b19d: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [24ms] sending RPC to server 993f936550594542941a497816c4b19d, [27ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [27ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@701958279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@5f5e4a7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f4a124 -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286))) after lookup
20:04:07.131 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:07.133 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.133 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.133 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.168 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:07.169 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:07.170 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [3ms] received response from server 993f936550594542941a497816c4b19d: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [24ms] sending RPC to server 993f936550594542941a497816c4b19d, [27ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [27ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 993f936550594542941a497816c4b19d, [47ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [47ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@701958279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@5f5e4a7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f4a124 -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592))) after lookup
20:04:07.171 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:07.173 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.174 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.174 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.308 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:07.309 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:07.309 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=223), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [3ms] received response from server 993f936550594542941a497816c4b19d: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [24ms] sending RPC to server 993f936550594542941a497816c4b19d, [27ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [27ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 993f936550594542941a497816c4b19d, [47ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [47ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [85ms] sending RPC to server 993f936550594542941a497816c4b19d, [88ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [88ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [217ms] refreshing cache from master, [217ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [222ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@701958279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@5f5e4a7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592) -> (continuation of Deferred@1902946782 after retry RPC@1467878441), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f4a124 -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592) -> (continuation of Deferred@1902946782 after retry RPC@1467878441))) after lookup
20:04:07.310 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:07.312 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.313 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.313 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.697 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:07.697 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:07.699 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=612), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [3ms] received response from server 993f936550594542941a497816c4b19d: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [24ms] sending RPC to server 993f936550594542941a497816c4b19d, [27ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [27ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 993f936550594542941a497816c4b19d, [47ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [47ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [85ms] sending RPC to server 993f936550594542941a497816c4b19d, [88ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [88ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [217ms] refreshing cache from master, [217ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [222ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [224ms] sending RPC to server 993f936550594542941a497816c4b19d, [227ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [227ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [598ms] refreshing cache from master, [599ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [610ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@701958279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@5f5e4a7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592) -> (continuation of Deferred@1902946782 after retry RPC@1467878441) -> (continuation of Deferred@245480856 after retry RPC@1686129813), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f4a124 -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592) -> (continuation of Deferred@1902946782 after retry RPC@1467878441) -> (continuation of Deferred@245480856 after retry RPC@1686129813))) after lookup
20:04:07.700 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:07.705 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.705 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:07.706 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:08.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.439821  7003 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 993f936550594542941a497816c4b19d)
20:04:08.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.439816  7002 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 993f936550594542941a497816c4b19d)
20:04:08.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.440353  7002 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:08.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.440374  7003 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:08.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.443017  7002 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:08.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.443372  6730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:08.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.444307  6730 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 1.
20:04:08.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.445001  7003 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:08.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.445744  6805 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482; no voters: 
20:04:08.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:08.446609  6664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:08.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.447039  7002 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Leader pre-election won for term 2
20:04:08.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.447405  7002 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 993f936550594542941a497816c4b19d)
20:04:08.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.447788  7002 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 1 FOLLOWER]: Advancing to term 2
20:04:08.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.447649  6870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a" is_pre_election: true
20:04:08.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:08.448899  6804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:08.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:08.453277  6664 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.455405  7002 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:08.455835  6804 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.456611  6870 raft_consensus.cc:2393] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e35e6d7e70184db4b58023c7bcd30482 in current term 2: Already voted for candidate 31b9b4da6f9945578c783b9a7a597b5a in this term.
20:04:08.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.457479  7002 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 2 election: Requested vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:08.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.458324  6730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:08.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.458930  6730 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 1 FOLLOWER]: Advancing to term 2
20:04:08.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.458982  6663 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d
20:04:08.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:08.462069  6804 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:08.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.465163  6730 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 2.
20:04:08.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.465646  7003 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
20:04:08.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.466387  6805 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482; no voters: 993f936550594542941a497816c4b19d
20:04:08.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.467324  7002 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 2 FOLLOWER]: Leader election won for term 2
20:04:08.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.468633  7002 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 2 LEADER]: Becoming Leader. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Running, Role: LEADER
20:04:08.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.469759  7002 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:08.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.479899  6398 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a reported cstate change: term changed from 1 to 2, leader changed from 993f936550594542941a497816c4b19d (127.6.14.65) to 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67). New cstate: current_term: 2 leader_uuid: "31b9b4da6f9945578c783b9a7a597b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: UNKNOWN } } }
20:04:08.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.855281  6730 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Refusing update from remote peer 31b9b4da6f9945578c783b9a7a597b5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
20:04:08.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:08.857149  7002 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
20:04:08.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:08.884855  6804 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a -> Peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Couldn't send request to peer 993f936550594542941a497816c4b19d. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:09.190 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:09.191 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:09.192 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2105), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [3ms] received response from server 993f936550594542941a497816c4b19d: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [24ms] sending RPC to server 993f936550594542941a497816c4b19d, [27ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [27ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 993f936550594542941a497816c4b19d, [47ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [47ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [85ms] sending RPC to server 993f936550594542941a497816c4b19d, [88ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [88ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [217ms] refreshing cache from master, [217ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [222ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [224ms] sending RPC to server 993f936550594542941a497816c4b19d, [227ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [227ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [598ms] refreshing cache from master, [599ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [610ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [614ms] sending RPC to server 993f936550594542941a497816c4b19d, [620ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [620ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [2097ms] refreshing cache from master, [2097ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [2103ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@701958279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@5f5e4a7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592) -> (continuation of Deferred@1902946782 after retry RPC@1467878441) -> (continuation of Deferred@245480856 after retry RPC@1686129813) -> (continuation of Deferred@1086216178 after retry RPC@649879444), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f4a124 -> wakeup thread Time-limited test -> (continuation of Deferred@875102300 after retry RPC@324639286) -> (continuation of Deferred@415895746 after retry RPC@463433592) -> (continuation of Deferred@1902946782 after retry RPC@1467878441) -> (continuation of Deferred@245480856 after retry RPC@1686129813) -> (continuation of Deferred@1086216178 after retry RPC@649879444))) after lookup
20:04:09.193 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:09.194 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] Successfully connected to peer
20:04:09.230 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x39092639, L:/127.0.0.1:41830 - R:/127.6.14.67:33469] using TOKEN/null
20:04:09.290 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.65:40343
20:04:09.295 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

20:04:09.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:09.298309  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:09.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:09.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:09.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:40343
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:04:09.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:09.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44657
20:04:09.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:04:09.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:09.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:09.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:09.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:09.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:09.675917  7029 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:09.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:09.676625  7029 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:09.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:09.677358  7029 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:09.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:09.737375  7029 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:09.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:09.738598  7029 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.65
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:09.782562  7029 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:40343
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:09.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44657
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:09.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:09.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:09.784914  7029 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:09.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:09.787098  7029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:09.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:09.803633  7036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:11.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:11.531288  6804 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a -> Peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Couldn't send request to peer 993f936550594542941a497816c4b19d. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
20:04:11.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:11.213621  7035 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 7029
20:04:11.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:11.628950  7035 kernel_stack_watchdog.cc:198] Thread 7029 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
20:04:11.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
20:04:11.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
20:04:11.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
20:04:11.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
20:04:11.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:11.629978  7029 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.827s	user 0.680s	sys 1.018s
20:04:11.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:11.630455  7029 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.828s	user 0.680s	sys 1.018s
20:04:11.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:09.803893  7037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:11.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:11.633203  7039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:11.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:11.636660  7038 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1829 milliseconds
20:04:11.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.636698  7029 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:11.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.638202  7029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:11.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.640930  7029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:11.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.642431  7029 hybrid_clock.cc:648] HybridClock initialized: now 1779134651642365 us; error 69 us; skew 500 ppm
20:04:11.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.643527  7029 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:11.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.651885  7029 webserver.cc:533] Webserver started at http://127.6.14.65:44657/ using document root <none> and password file <none>
20:04:11.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.653167  7029 fs_manager.cc:362] Metadata directory not provided
20:04:11.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.653493  7029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:11.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.663734  7029 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.005s	sys 0.001s
20:04:11.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.669854  7053 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:11.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.671214  7029 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
20:04:11.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.671761  7029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:11.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "993f936550594542941a497816c4b19d"
20:04:11.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:58 on dist-test-slave-vt9x"
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.674443  7029 fs_report.cc:389] FS layout report
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/data
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:11.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.735158  7029 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:11.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.737310  7029 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:11.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.737974  7029 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:11.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.741696  7029 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:11.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.749167  7060 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:11.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.759235  7029 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:11.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.759582  7029 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.000s	sys 0.003s
20:04:11.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.760092  7029 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:11.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.766064  7029 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:11.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.766378  7029 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.001s	sys 0.003s
20:04:11.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.766893  7060 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap starting.
20:04:11.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.831033  7060 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Log is configured to *not* fsync() on all Append() calls
20:04:11.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.869930  7060 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:11.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.871933  7060 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap complete.
20:04:11.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.874424  7060 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent bootstrapping tablet: real 0.108s	user 0.052s	sys 0.049s
20:04:11.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.910040  7060 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:11.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.914865  7060 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993f936550594542941a497816c4b19d, State: Initialized, Role: FOLLOWER
20:04:11.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.917186  7060 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:11.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.921644  7060 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent starting tablet: real 0.047s	user 0.041s	sys 0.005s
20:04:11.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.983273  7029 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.65:40343
20:04:11.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.983806  7168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.65:40343 every 8 connection(s)
20:04:11.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.986315  7029 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:11.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:11.989764  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 7029
20:04:11.990 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:12.041 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:12.043 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.67:33469
20:04:12.044 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:12.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.044559  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 6783
20:04:12.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.057322  7169 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:12.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.057400  7170 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:12.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.058063  7169 heartbeater.cc:461] Registering TS with master...
20:04:12.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.058135  7170 heartbeater.cc:461] Registering TS with master...
20:04:12.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.059808  7171 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:12.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.059918  7170 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:12.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.060220  7171 heartbeater.cc:461] Registering TS with master...
20:04:12.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.059803  7169 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:12.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.061208  7171 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:12.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.066011  6397 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:12.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.066457  6323 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:12.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.069281  6397 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.65:32935
20:04:12.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:12.075958  6246 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:12.089 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:12.089 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] handling channelInactive
20:04:12.090 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state READY due to: connection closed
20:04:12.091 [DEBUG - Time-limited test] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:12.092 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.093 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.093 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.110 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:12.110 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:12.111 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1596412653(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@20f9d27c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e2ad5ca -> wakeup thread Time-limited test)) after lookup
20:04:12.112 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:12.114 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.114 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.114 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.131 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:12.132 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:12.132 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [22ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [24ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [24ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1596412653(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@20f9d27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e2ad5ca -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223))) after lookup
20:04:12.134 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:12.135 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.135 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.136 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.149 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:12.149 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:12.149 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [22ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [24ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [24ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [43ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [46ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [58ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1596412653(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@20f9d27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e2ad5ca -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371))) after lookup
20:04:12.150 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:12.151 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.151 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.152 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.209 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:12.210 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:12.210 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=120), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [22ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [24ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [24ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [43ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [46ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [58ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [60ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [61ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [62ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [113ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [118ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1596412653(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@20f9d27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371) -> (continuation of Deferred@784018505 after retry RPC@2080434540), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e2ad5ca -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371) -> (continuation of Deferred@784018505 after retry RPC@2080434540))) after lookup
20:04:12.212 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:12.213 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.213 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.213 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:12.525892  6354 debug-util.cc:398] Leaking SignalData structure 0x7b080008ac80 after lost signal to thread 6287
20:04:12.689 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:12.690 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:12.690 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=600), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [22ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [24ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [24ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [43ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [46ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [58ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [60ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [61ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [62ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [113ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [118ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [121ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [123ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [123ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [593ms] refreshing cache from master, [593ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [599ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1596412653(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@20f9d27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371) -> (continuation of Deferred@784018505 after retry RPC@2080434540) -> (continuation of Deferred@448009092 after retry RPC@1636090423), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e2ad5ca -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371) -> (continuation of Deferred@784018505 after retry RPC@2080434540) -> (continuation of Deferred@448009092 after retry RPC@1636090423))) after lookup
20:04:12.691 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:12.693 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.694 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:12.694 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:13.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.073803  7169 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:13.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.370718  7182 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:13.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.371497  7182 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:13.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.374327  7182 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:13.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:13.388636  6663 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:13.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.398506  7123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d" is_pre_election: true
20:04:13.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.399488  7123 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35e6d7e70184db4b58023c7bcd30482 in term 1.
20:04:13.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.401511  6664 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:13.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.402680  7182 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Leader pre-election won for term 3
20:04:13.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.403185  7182 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:13.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.403615  7182 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 2 FOLLOWER]: Advancing to term 3
20:04:13.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.409754  7182 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:13.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.411520  7182 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 3 election: Requested vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:13.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.415093  7123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d"
20:04:13.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.415799  7123 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 1 FOLLOWER]: Advancing to term 3
20:04:13.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:13.415545  6663 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:13.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.425937  7123 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35e6d7e70184db4b58023c7bcd30482 in term 3.
20:04:13.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.427287  6664 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:13.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.428119  7182 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 3 FOLLOWER]: Leader election won for term 3
20:04:13.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.429764  7182 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 3 LEADER]: Becoming Leader. State: Replica: e35e6d7e70184db4b58023c7bcd30482, State: Running, Role: LEADER
20:04:13.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.431211  7182 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:13.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.444314  6397 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 reported cstate change: term changed from 2 to 3, leader changed from 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67) to e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66). New cstate: current_term: 3 leader_uuid: "e35e6d7e70184db4b58023c7bcd30482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: UNKNOWN } } }
20:04:13.790 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:13.790 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:13.791 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1701), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [22ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [24ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [24ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [43ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [46ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [58ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [60ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [61ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [62ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [113ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [118ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [121ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [123ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [123ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [593ms] refreshing cache from master, [593ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [599ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [601ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [604ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [604ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [1694ms] refreshing cache from master, [1694ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1699ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1596412653(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@20f9d27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371) -> (continuation of Deferred@784018505 after retry RPC@2080434540) -> (continuation of Deferred@448009092 after retry RPC@1636090423) -> (continuation of Deferred@1928743821 after retry RPC@1247157007), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1e2ad5ca -> wakeup thread Time-limited test -> (continuation of Deferred@1686153304 after retry RPC@751609223) -> (continuation of Deferred@71998680 after retry RPC@1298391371) -> (continuation of Deferred@784018505 after retry RPC@2080434540) -> (continuation of Deferred@448009092 after retry RPC@1636090423) -> (continuation of Deferred@1928743821 after retry RPC@1247157007))) after lookup
20:04:13.792 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:13.794 [DEBUG - kudu-nio-0] (Connection.java:757) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] Successfully connected to peer
20:04:13.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.821779  7123 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 3 FOLLOWER]: Refusing update from remote peer e35e6d7e70184db4b58023c7bcd30482: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
20:04:13.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.823988  7182 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
20:04:13.843 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x9e943ba6, L:/127.0.0.1:40224 - R:/127.6.14.66:36169] using TOKEN/null
20:04:13.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:13.868075  6663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111) [suppressed 2 similar messages]
20:04:13.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:13.872172  6663 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 -> Peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Couldn't send request to peer 31b9b4da6f9945578c783b9a7a597b5a. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:13.957 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.67:33469
20:04:13.958 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:13.960440  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:13.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:13.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:14.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:14.394639  7209 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:14.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:14.395391  7209 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:14.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:14.396136  7209 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:14.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:14.439337  7209 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:14.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:14.440645  7209 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.67
20:04:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:14.492069  7209 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:14.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:14.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:14.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:14.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:14.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:14.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:14.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:14.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:14.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:14.494283  7209 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:14.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:14.496465  7209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:14.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:14.518035  7215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:15.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:14.521152  7216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:15.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:15.855887  7209 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.337s	user 0.514s	sys 0.801s
20:04:15.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:15.856243  7209 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.337s	user 0.514s	sys 0.801s
20:04:15.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.858141  7209 server_base.cc:1061] running on GCE node
20:04:15.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:15.860288  7224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:15.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.861778  7209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:15.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.864739  7209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:15.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.866227  7209 hybrid_clock.cc:648] HybridClock initialized: now 1779134655866172 us; error 63 us; skew 500 ppm
20:04:15.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.867435  7209 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.876473  7209 webserver.cc:533] Webserver started at http://127.6.14.67:45941/ using document root <none> and password file <none>
20:04:15.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.877700  7209 fs_manager.cc:362] Metadata directory not provided
20:04:15.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.877993  7209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.888360  7209 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.005s	sys 0.001s
20:04:15.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.894614  7229 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:15.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.896204  7209 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.003s
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.896625  7209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:03 on dist-test-slave-vt9x"
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.899036  7209 fs_report.cc:389] FS layout report
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/data
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:15.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:15.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.986409  7209 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:15.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.989380  7209 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:15.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.990010  7209 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:15.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:15.993790  7209 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:16.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.001577  7236 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:16.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.015628  7209 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:16.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.015974  7209 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.017s	user 0.001s	sys 0.002s
20:04:16.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.016330  7209 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:16.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.021855  7209 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:16.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.022096  7209 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.002s	sys 0.002s
20:04:16.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.022709  7236 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap starting.
20:04:16.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.089754  7236 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Log is configured to *not* fsync() on all Append() calls
20:04:16.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.139408  7236 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:16.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.141026  7236 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap complete.
20:04:16.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.143311  7236 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent bootstrapping tablet: real 0.121s	user 0.067s	sys 0.050s
20:04:16.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.174407  7236 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:16.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.179273  7236 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Initialized, Role: FOLLOWER
20:04:16.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.181320  7236 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:16.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.187181  7236 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent starting tablet: real 0.043s	user 0.024s	sys 0.015s
20:04:16.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.245419  7209 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.67:33469
20:04:16.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.245944  7344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.67:33469 every 8 connection(s)
20:04:16.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.249289  7209 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:16.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.253888  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 7209
20:04:16.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:16.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.277832  7299 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 2 FOLLOWER]: Advancing to term 3
20:04:16.292 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:16.293 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.66:36169
20:04:16.294 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

20:04:16.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.295065  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 6645
20:04:16.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.330272  7348 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:16.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.330968  7348 heartbeater.cc:461] Registering TS with master...
20:04:16.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.332263  7347 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:16.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.332633  7347 heartbeater.cc:461] Registering TS with master...
20:04:16.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.333374  7348 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:16.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.333638  7347 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:16.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.334618  7346 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:16.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.335019  7346 heartbeater.cc:461] Registering TS with master...
20:04:16.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.336002  7346 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:16.344 [DEBUG - kudu-nio-0] (Connection.java:239) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] handling channelInactive
20:04:16.345 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state READY due to: connection closed
20:04:16.347 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:16.348 [DEBUG - Time-limited test] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:16.350 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.351 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.351 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:16.354501  7231 connection.cc:465] server connection from 127.6.14.66:37891 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.14.66:37891 (request call id 0) could send its response
20:04:16.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.361563  6398 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:16.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.361847  6246 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:16.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.362054  6323 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:16.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.365243  6398 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.67:50595
20:04:16.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:16.369814  7346 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:16.371 [DEBUG - kudu-nio-1] (NetUtil.java:139) Resolved '127.6.14.66' into [/127.6.14.66] in 1337723ns
20:04:16.371 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:16.372 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:16.372 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [3ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [4ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [21ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1609833085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@166fbdff -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ecf050 -> wakeup thread Time-limited test)) after lookup
20:04:16.373 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:16.375 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.375 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.375 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.388 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:16.388 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:16.389 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [3ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [4ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [21ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [25ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [27ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [27ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1609833085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@166fbdff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ecf050 -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866))) after lookup
20:04:16.389 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:16.390 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.391 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.391 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.429 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:16.430 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:16.431 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [3ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [4ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [21ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [25ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [27ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [27ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [41ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [43ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [43ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [81ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1609833085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@166fbdff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ecf050 -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498))) after lookup
20:04:16.432 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:16.433 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.433 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.433 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.529 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:16.529 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:16.530 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=182), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [3ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [4ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [21ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [25ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [27ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [27ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [41ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [43ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [43ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [81ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [83ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [85ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [85ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [180ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1609833085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@166fbdff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ecf050 -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263))) after lookup
20:04:16.531 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:16.532 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.533 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.533 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.909 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:16.909 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:16.910 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=562), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [3ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [4ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [21ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [25ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [27ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [27ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [41ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [43ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [43ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [81ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [83ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [85ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [85ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [180ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [183ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [185ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [185ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [555ms] refreshing cache from master, [555ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [560ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1609833085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@166fbdff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263) -> (continuation of Deferred@81807432 after retry RPC@866421129), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ecf050 -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263) -> (continuation of Deferred@81807432 after retry RPC@866421129))) after lookup
20:04:16.911 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:16.912 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.913 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:16.913 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:17.489 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:17.489 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:17.490 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1142), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [3ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [4ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [21ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [25ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [27ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [27ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [41ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [43ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [43ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [81ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [83ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [85ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [85ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [180ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [183ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [185ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [185ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [555ms] refreshing cache from master, [555ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [560ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [563ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [565ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [565ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [1135ms] refreshing cache from master, [1136ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1140ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1609833085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@166fbdff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263) -> (continuation of Deferred@81807432 after retry RPC@866421129) -> (continuation of Deferred@1712249491 after retry RPC@1903292186), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ecf050 -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263) -> (continuation of Deferred@81807432 after retry RPC@866421129) -> (continuation of Deferred@1712249491 after retry RPC@1903292186))) after lookup
20:04:17.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.490571  7361 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e35e6d7e70184db4b58023c7bcd30482)
20:04:17.491 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:17.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.491194  7361 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:17.492 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:17.492 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:17.493 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:17.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.494549  7361 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:17.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:17.499056  7056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:17.503247  7056 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:17.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.506654  7299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a" is_pre_election: true
20:04:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.508639  7054 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993f936550594542941a497816c4b19d; no voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482
20:04:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.509490  7361 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
20:04:17.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.819640  7367 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e35e6d7e70184db4b58023c7bcd30482)
20:04:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.820339  7367 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:17.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.823146  7367 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:17.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:17.828008  7232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:17.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:17.832902  7232 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:17.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.836628  7123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d" is_pre_election: true
20:04:17.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.837262  7123 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 3.
20:04:17.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.838505  7231 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d; no voters: e35e6d7e70184db4b58023c7bcd30482
20:04:17.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.839406  7367 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 3 FOLLOWER]: Leader pre-election won for term 4
20:04:17.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.839767  7367 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e35e6d7e70184db4b58023c7bcd30482)
20:04:17.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.840022  7367 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 3 FOLLOWER]: Advancing to term 4
20:04:17.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.845652  7367 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:17.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.847486  7367 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 4 election: Requested vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:17.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.848945  7123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d"
20:04:17.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.849606  7123 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 3 FOLLOWER]: Advancing to term 4
20:04:17.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:17.851231  7232 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.854812  7123 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 4.
20:04:17.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.855882  7231 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d; no voters: e35e6d7e70184db4b58023c7bcd30482
20:04:17.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.856575  7367 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Leader election won for term 4
20:04:17.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.858392  7367 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 LEADER]: Becoming Leader. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Running, Role: LEADER
20:04:17.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.859886  7367 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:17.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:17.869918  6397 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a reported cstate change: term changed from 3 to 4, leader changed from e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66) to 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67). New cstate: current_term: 4 leader_uuid: "31b9b4da6f9945578c783b9a7a597b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: UNKNOWN } } }
20:04:18.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:18.262419  7123 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 4 FOLLOWER]: Refusing update from remote peer 31b9b4da6f9945578c783b9a7a597b5a: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
20:04:18.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:18.264153  7367 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
20:04:18.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:18.289341  7232 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a -> Peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Couldn't send request to peer e35e6d7e70184db4b58023c7bcd30482. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:18.570 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:18.571 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:18.573 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2225), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [3ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [4ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [21ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [25ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [27ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [27ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [40ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [41ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [43ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [43ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [81ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [83ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [85ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [85ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [180ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [183ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [185ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [185ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [555ms] refreshing cache from master, [555ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [560ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [563ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [565ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [565ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [1135ms] refreshing cache from master, [1136ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1140ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [1142ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [1144ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [1145ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2215ms] refreshing cache from master, [2215ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [2222ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1609833085(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@166fbdff -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263) -> (continuation of Deferred@81807432 after retry RPC@866421129) -> (continuation of Deferred@1712249491 after retry RPC@1903292186) -> (continuation of Deferred@1412461574 after retry RPC@717703333), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2ecf050 -> wakeup thread Time-limited test -> (continuation of Deferred@635878534 after retry RPC@1080931866) -> (continuation of Deferred@1770199654 after retry RPC@1148435498) -> (continuation of Deferred@1818962212 after retry RPC@899697263) -> (continuation of Deferred@81807432 after retry RPC@866421129) -> (continuation of Deferred@1712249491 after retry RPC@1903292186) -> (continuation of Deferred@1412461574 after retry RPC@717703333))) after lookup
20:04:18.575 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:18.576 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] Successfully connected to peer
20:04:18.607 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xffa4a145, L:/127.0.0.1:58258 - R:/127.6.14.67:33469] using TOKEN/null
20:04:18.661 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.66:36169
20:04:18.662 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

20:04:18.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:18.664259  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:18.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:18.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:36169
20:04:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:04:18.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:18.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41631
20:04:18.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:04:18.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:18.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:18.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:18.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:19.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.042026  7394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:19.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.042613  7394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:19.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.043097  7394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:19.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.080715  7394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:19.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.081625  7394 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.66
20:04:19.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:19.123234  7394 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:36169
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41631
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:19.125015  7394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:19.126878  7394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:19.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.142292  7403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:20.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.147254  7406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:20.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:19.143209  7404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:20.515697  7405 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1367 milliseconds
20:04:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.515857  7394 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:20.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.517452  7394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:20.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.520699  7394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:20.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.522173  7394 hybrid_clock.cc:648] HybridClock initialized: now 1779134660522126 us; error 73 us; skew 500 ppm
20:04:20.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.523053  7394 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:20.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.531136  7394 webserver.cc:533] Webserver started at http://127.6.14.66:41631/ using document root <none> and password file <none>
20:04:20.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.532286  7394 fs_manager.cc:362] Metadata directory not provided
20:04:20.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.532522  7394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:20.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.541431  7394 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.000s	sys 0.004s
20:04:20.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.546844  7418 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:20.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.548172  7394 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
20:04:20.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.548629  7394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:20.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:20.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:01 on dist-test-slave-vt9x"
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.550843  7394 fs_report.cc:389] FS layout report
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/data
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:20.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:20.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:20.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:20.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:20.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.622145  7394 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:20.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.624001  7394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:20.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.624454  7394 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:20.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.627398  7394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:20.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.634327  7425 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:20.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.644359  7394 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:20.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.644623  7394 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.001s
20:04:20.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.644939  7394 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:20.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.650297  7394 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:20.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.650508  7394 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.000s	sys 0.003s
20:04:20.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.651297  7425 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap starting.
20:04:20.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.731256  7425 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Log is configured to *not* fsync() on all Append() calls
20:04:20.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.818470  7425 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:20.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.819958  7425 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap complete.
20:04:20.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.822396  7425 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent bootstrapping tablet: real 0.172s	user 0.123s	sys 0.038s
20:04:20.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:20.859591  7232 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a -> Peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Couldn't send request to peer e35e6d7e70184db4b58023c7bcd30482. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
20:04:20.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.858129  7425 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:20.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.862237  7425 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35e6d7e70184db4b58023c7bcd30482, State: Initialized, Role: FOLLOWER
20:04:20.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.863639  7425 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:20.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.869664  7425 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent starting tablet: real 0.047s	user 0.040s	sys 0.009s
20:04:20.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.900643  7394 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.66:36169
20:04:20.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.901021  7537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.66:36169 every 8 connection(s)
20:04:20.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.903445  7394 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:20.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.910303  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 7394
20:04:20.911 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:20.947 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:20.955 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.67:33469
20:04:20.956 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:20.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.956794  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 7209
20:04:20.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.956734  7539 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:20.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.957346  7539 heartbeater.cc:461] Registering TS with master...
20:04:20.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.958990  7539 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:20.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.959678  7538 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:20.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.960057  7538 heartbeater.cc:461] Registering TS with master...
20:04:20.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.961091  7538 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:20.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.964558  6398 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:20.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.964934  6323 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:20.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.967720  6398 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.66:47639
20:04:20.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.971623  7540 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:20.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.972071  7540 heartbeater.cc:461] Registering TS with master...
20:04:20.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.972846  7540 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:20.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:20.977074  6245 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:21.014 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] handling channelInactive
20:04:21.015 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state READY due to: connection closed
20:04:21.015 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:21.016 [DEBUG - Time-limited test] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:21.018 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.018 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.018 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.029 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:21.029 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:21.030 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@797880523(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@6e43b82b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@466770c8 -> wakeup thread Time-limited test)) after lookup
20:04:21.030 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:21.031 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.032 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.032 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.049 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:21.049 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:21.049 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [14ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [16ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [16ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [32ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@797880523(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@6e43b82b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@466770c8 -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846))) after lookup
20:04:21.051 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:21.053 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.053 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.053 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.088 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:21.089 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:21.089 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [14ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [16ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [16ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [32ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [34ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [37ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [72ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@797880523(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@6e43b82b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@466770c8 -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518))) after lookup
20:04:21.090 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:21.091 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.091 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.092 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.229 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:21.229 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:21.230 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=214), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [14ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [16ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [16ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [32ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [34ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [37ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [72ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [74ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [76ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [76ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [207ms] refreshing cache from master, [208ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [212ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@797880523(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@6e43b82b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@466770c8 -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302))) after lookup
20:04:21.231 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:21.233 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.233 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.233 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.529 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:21.529 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:21.530 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=514), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [14ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [16ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [16ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [32ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [34ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [37ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [72ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [74ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [76ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [76ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [207ms] refreshing cache from master, [208ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [212ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [215ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [217ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [217ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [507ms] refreshing cache from master, [508ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [512ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@797880523(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@6e43b82b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302) -> (continuation of Deferred@1376810760 after retry RPC@109792306), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@466770c8 -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302) -> (continuation of Deferred@1376810760 after retry RPC@109792306))) after lookup
20:04:21.531 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:21.532 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.533 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.533 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:21.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:21.973227  7538 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:22.129 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:22.130 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:22.130 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1114), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [14ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [16ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [16ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [32ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [34ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [37ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [72ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [74ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [76ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [76ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [207ms] refreshing cache from master, [208ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [212ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [215ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [217ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [217ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [507ms] refreshing cache from master, [508ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [512ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [514ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [517ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [517ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [1107ms] refreshing cache from master, [1108ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1113ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@797880523(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@6e43b82b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302) -> (continuation of Deferred@1376810760 after retry RPC@109792306) -> (continuation of Deferred@936430658 after retry RPC@1801907286), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@466770c8 -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302) -> (continuation of Deferred@1376810760 after retry RPC@109792306) -> (continuation of Deferred@936430658 after retry RPC@1801907286))) after lookup
20:04:22.131 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:22.132 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:22.132 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:22.132 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:22.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.177083  7549 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:22.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.177706  7549 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:22.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.179737  7549 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:22.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:22.188702  7054 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:22.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.194281  7489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:22.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.195072  7489 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 3.
20:04:22.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.196657  7056 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:22.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.197458  7549 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 4 FOLLOWER]: Leader pre-election won for term 5
20:04:22.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.197779  7549 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:22.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.198123  7549 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 4 FOLLOWER]: Advancing to term 5
20:04:22.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.203522  7549 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.205224  7549 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 5 election: Requested vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:22.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.206477  7489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:22.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.207159  7489 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 3 FOLLOWER]: Advancing to term 5
20:04:22.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:22.209926  7054 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:22.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.215092  7489 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 5.
20:04:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.216166  7056 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:22.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.216832  7549 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Leader election won for term 5
20:04:22.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.218152  7549 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 LEADER]: Becoming Leader. State: Replica: 993f936550594542941a497816c4b19d, State: Running, Role: LEADER
20:04:22.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.219455  7549 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:22.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.230420  6398 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d reported cstate change: term changed from 4 to 5, leader changed from 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67) to 993f936550594542941a497816c4b19d (127.6.14.65). New cstate: current_term: 5 leader_uuid: "993f936550594542941a497816c4b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: HEALTHY } } }
20:04:22.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:22.782948  7054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111) [suppressed 2 similar messages]
20:04:22.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:22.786087  7054 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d -> Peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Couldn't send request to peer 31b9b4da6f9945578c783b9a7a597b5a. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:22.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.838182  7489 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 5 FOLLOWER]: Refusing update from remote peer 993f936550594542941a497816c4b19d: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
20:04:22.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:22.839864  7566 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
20:04:24.249 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:24.249 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:24.250 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=3233), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [13ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [14ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [16ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [16ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [32ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [34ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [37ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [72ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [74ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [76ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [76ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [207ms] refreshing cache from master, [208ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [212ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [215ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [217ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [217ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [507ms] refreshing cache from master, [508ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [512ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [514ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [517ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [517ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [1107ms] refreshing cache from master, [1108ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1113ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [1114ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [1116ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [1116ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3227ms] refreshing cache from master, [3227ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [3232ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@797880523(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@6e43b82b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302) -> (continuation of Deferred@1376810760 after retry RPC@109792306) -> (continuation of Deferred@936430658 after retry RPC@1801907286) -> (continuation of Deferred@1701353807 after retry RPC@967892456), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@466770c8 -> wakeup thread Time-limited test -> (continuation of Deferred@1079086732 after retry RPC@1028478846) -> (continuation of Deferred@1020506789 after retry RPC@1811084518) -> (continuation of Deferred@1444304374 after retry RPC@717217302) -> (continuation of Deferred@1376810760 after retry RPC@109792306) -> (continuation of Deferred@936430658 after retry RPC@1801907286) -> (continuation of Deferred@1701353807 after retry RPC@967892456))) after lookup
20:04:24.251 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:24.252 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] Successfully connected to peer
20:04:24.274 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x2fdbe0ab, L:/127.0.0.1:34712 - R:/127.6.14.65:40343] using TOKEN/null
20:04:24.327 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.67:33469
20:04:24.328 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:24.330102  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:24.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:24.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:24.700706  7588 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:24.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:24.701272  7588 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:24.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:24.701756  7588 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:24.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:24.742050  7588 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:24.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:24.743449  7588 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.67
20:04:24.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:24.789274  7588 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:24.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:24.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:24.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:24.791050  7588 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:24.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:24.793087  7588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:24.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:24.807649  7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:25.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:25.187822  7054 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d -> Peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Couldn't send request to peer 31b9b4da6f9945578c783b9a7a597b5a. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
20:04:26.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:24.809007  7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:26.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:26.211485  7595 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 7588
20:04:26.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:26.259865  7595 kernel_stack_watchdog.cc:198] Thread 7588 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
20:04:26.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
20:04:26.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
20:04:26.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
20:04:26.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
20:04:26.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:26.266141  7588 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.457s	user 0.003s	sys 0.005s
20:04:26.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:26.266598  7588 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.458s	user 0.003s	sys 0.005s
20:04:26.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:26.266598  7598 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1450 milliseconds
20:04:26.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:26.268530  7599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:26.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.268630  7588 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:26.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.269963  7588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:26.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.272893  7588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:26.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.274343  7588 hybrid_clock.cc:648] HybridClock initialized: now 1779134666274296 us; error 58 us; skew 500 ppm
20:04:26.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.275247  7588 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:26.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.282605  7588 webserver.cc:533] Webserver started at http://127.6.14.67:45941/ using document root <none> and password file <none>
20:04:26.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.283778  7588 fs_manager.cc:362] Metadata directory not provided
20:04:26.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.283977  7588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:26.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.292778  7588 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.001s
20:04:26.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.297974  7610 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:26.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.299506  7588 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.299953  7588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:03 on dist-test-slave-vt9x"
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.302129  7588 fs_report.cc:389] FS layout report
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/data
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:26.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:26.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:26.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:26.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:26.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.353266  7588 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:26.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.354995  7588 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:26.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.355522  7588 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:26.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.358634  7588 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:26.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.365391  7617 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:26.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.374907  7588 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:26.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.375197  7588 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
20:04:26.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.375540  7588 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:26.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.381445  7588 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:26.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.381739  7588 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
20:04:26.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.382231  7617 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap starting.
20:04:26.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.447299  7617 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Log is configured to *not* fsync() on all Append() calls
20:04:26.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.517598  7617 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:26.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.518764  7617 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap complete.
20:04:26.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.520789  7617 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent bootstrapping tablet: real 0.139s	user 0.083s	sys 0.053s
20:04:26.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.543283  7617 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:26.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.546509  7617 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Initialized, Role: FOLLOWER
20:04:26.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.547928  7617 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:26.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.552405  7617 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent starting tablet: real 0.031s	user 0.026s	sys 0.006s
20:04:26.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.598330  7588 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.67:33469
20:04:26.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.598853  7725 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.67:33469 every 8 connection(s)
20:04:26.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.601478  7588 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:26.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.611196  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 7588
20:04:26.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:26.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.644649  7726 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:26.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.645314  7726 heartbeater.cc:461] Registering TS with master...
20:04:26.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.646261  7727 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:26.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.646632  7727 heartbeater.cc:461] Registering TS with master...
20:04:26.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.646950  7726 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:26.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.649044  7727 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:26.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.653626  6398 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:26.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.653838  6322 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:26.657 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:26.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.657213  6398 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.67:54215
20:04:26.660 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.65:40343
20:04:26.660 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

20:04:26.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.661051  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 7029
20:04:26.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.665786  7728 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:26.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.666257  7728 heartbeater.cc:461] Registering TS with master...
20:04:26.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.667054  7728 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:26.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:26.670629  6245 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:26.697 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] handling channelInactive
20:04:26.698 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state READY due to: connection closed
20:04:26.698 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:26.700 [DEBUG - Time-limited test] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:26.701 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.701 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.702 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.709 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:26.709 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:26.710 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1524902084(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@160f40ca -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6112d53e -> wakeup thread Time-limited test)) after lookup
20:04:26.711 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:26.712 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.712 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.713 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.728 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:26.729 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:26.729 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server 993f936550594542941a497816c4b19d, [13ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [14ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1524902084(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@160f40ca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6112d53e -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727))) after lookup
20:04:26.730 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:26.732 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.732 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.732 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.748 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:26.749 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:26.749 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server 993f936550594542941a497816c4b19d, [13ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [14ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [31ms] sending RPC to server 993f936550594542941a497816c4b19d, [33ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [33ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [49ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1524902084(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@160f40ca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6112d53e -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228))) after lookup
20:04:26.750 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:26.751 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.751 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.751 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.888 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:26.889 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:26.889 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server 993f936550594542941a497816c4b19d, [13ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [14ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [31ms] sending RPC to server 993f936550594542941a497816c4b19d, [33ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [33ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [49ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [50ms] sending RPC to server 993f936550594542941a497816c4b19d, [52ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [52ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [189ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1524902084(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@160f40ca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228) -> (continuation of Deferred@123355956 after retry RPC@1573970273), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6112d53e -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228) -> (continuation of Deferred@123355956 after retry RPC@1573970273))) after lookup
20:04:26.890 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:26.892 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.892 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:26.892 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:27.308 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:27.309 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:27.309 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=610), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server 993f936550594542941a497816c4b19d, [13ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [14ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [31ms] sending RPC to server 993f936550594542941a497816c4b19d, [33ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [33ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [49ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [50ms] sending RPC to server 993f936550594542941a497816c4b19d, [52ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [52ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [189ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [191ms] sending RPC to server 993f936550594542941a497816c4b19d, [193ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [193ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [604ms] refreshing cache from master, [604ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [609ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1524902084(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@160f40ca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228) -> (continuation of Deferred@123355956 after retry RPC@1573970273) -> (continuation of Deferred@708225962 after retry RPC@1882692708), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6112d53e -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228) -> (continuation of Deferred@123355956 after retry RPC@1573970273) -> (continuation of Deferred@708225962 after retry RPC@1882692708))) after lookup
20:04:27.310 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:27.311 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:27.312 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:27.312 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:27.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:27.665899  7726 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:27.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:27.904247  7737 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:04:27.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:27.904780  7737 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:27.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:27.907507  7737 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:27.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:27.919859  7612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:27.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:27.925082  7612 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:27.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:27.927362  7489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:27.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:27.929189  7613 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a; no voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482
20:04:27.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:27.930012  7737 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
20:04:28.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.084401  7742 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 993f936550594542941a497816c4b19d)
20:04:28.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.084983  7742 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:28.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.087792  7742 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:28.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:28.092265  7420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:28.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:28.097312  7420 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:28.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.101368  7680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a" is_pre_election: true
20:04:28.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.102126  7680 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35e6d7e70184db4b58023c7bcd30482 in term 4.
20:04:28.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.103775  7419 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482; no voters: 993f936550594542941a497816c4b19d
20:04:28.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.104607  7742 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 5 FOLLOWER]: Leader pre-election won for term 6
20:04:28.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.104935  7742 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 993f936550594542941a497816c4b19d)
20:04:28.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.105274  7742 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 5 FOLLOWER]: Advancing to term 6
20:04:28.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.110489  7742 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:28.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.112337  7742 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 6 election: Requested vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:28.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.113469  7680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:28.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.114007  7680 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 4 FOLLOWER]: Advancing to term 6
20:04:28.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:28.116326  7420 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:28.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.122258  7680 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35e6d7e70184db4b58023c7bcd30482 in term 6.
20:04:28.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.123337  7419 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482; no voters: 993f936550594542941a497816c4b19d
20:04:28.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.124202  7742 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 6 FOLLOWER]: Leader election won for term 6
20:04:28.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.126075  7742 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 6 LEADER]: Becoming Leader. State: Replica: e35e6d7e70184db4b58023c7bcd30482, State: Running, Role: LEADER
20:04:28.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.127117  7742 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:28.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.137395  6398 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 reported cstate change: term changed from 5 to 6, leader changed from 993f936550594542941a497816c4b19d (127.6.14.65) to e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66). New cstate: current_term: 6 leader_uuid: "e35e6d7e70184db4b58023c7bcd30482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: UNKNOWN } } }
20:04:28.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.568743  7680 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 6 FOLLOWER]: Refusing update from remote peer e35e6d7e70184db4b58023c7bcd30482: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
20:04:28.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.572026  7742 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
20:04:28.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:28.578967  7420 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 -> Peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Couldn't send request to peer 993f936550594542941a497816c4b19d. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:28.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:28.589986  7762 mvcc.cc:204] Tried to move back new op lower bound from 7287335584891084800 to 7287335576459055104. Current Snapshot: MvccSnapshot[applied={T|T < 7287335584891084800}]
20:04:29.008 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:29.009 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:29.009 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2310), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server 993f936550594542941a497816c4b19d, [13ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [14ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [31ms] sending RPC to server 993f936550594542941a497816c4b19d, [33ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [33ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [49ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [50ms] sending RPC to server 993f936550594542941a497816c4b19d, [52ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [52ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [189ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [191ms] sending RPC to server 993f936550594542941a497816c4b19d, [193ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [193ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [604ms] refreshing cache from master, [604ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [609ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [611ms] sending RPC to server 993f936550594542941a497816c4b19d, [613ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [613ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [2304ms] refreshing cache from master, [2304ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [2309ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1524902084(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@160f40ca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228) -> (continuation of Deferred@123355956 after retry RPC@1573970273) -> (continuation of Deferred@708225962 after retry RPC@1882692708) -> (continuation of Deferred@1933200511 after retry RPC@1067592384), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6112d53e -> wakeup thread Time-limited test -> (continuation of Deferred@940246917 after retry RPC@1330093727) -> (continuation of Deferred@419877909 after retry RPC@1782977228) -> (continuation of Deferred@123355956 after retry RPC@1573970273) -> (continuation of Deferred@708225962 after retry RPC@1882692708) -> (continuation of Deferred@1933200511 after retry RPC@1067592384))) after lookup
20:04:29.011 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:29.012 [DEBUG - kudu-nio-7] (Connection.java:757) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] Successfully connected to peer
20:04:29.058 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x5dea238b, L:/127.0.0.1:44966 - R:/127.6.14.66:36169] using TOKEN/null
20:04:29.127 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.65:40343
20:04:29.129 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:29.130986  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:40343
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44657
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:29.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:29.494511  7772 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:29.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:29.495082  7772 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:29.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:29.495576  7772 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:29.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:29.532840  7772 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:29.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:29.533931  7772 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.65
20:04:29.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:29.578337  7772 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:40343
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44657
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:29.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:29.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:29.580070  7772 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:29.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:29.582204  7772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:29.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:29.598928  7781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:31.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:31.335716  7420 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 -> Peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Couldn't send request to peer 993f936550594542941a497816c4b19d. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
20:04:31.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:31.005970  7779 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 7772
20:04:31.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:31.462262  7772 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.862s	user 0.680s	sys 1.145s
20:04:31.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:31.462740  7772 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.863s	user 0.680s	sys 1.145s
20:04:31.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:29.600275  7780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:31.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:31.465060  7783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:31.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:31.468436  7782 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1863 milliseconds
20:04:31.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.468473  7772 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:31.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.469965  7772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:31.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.472544  7772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:31.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.474105  7772 hybrid_clock.cc:648] HybridClock initialized: now 1779134671474047 us; error 63 us; skew 500 ppm
20:04:31.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.475124  7772 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:31.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.482630  7772 webserver.cc:533] Webserver started at http://127.6.14.65:44657/ using document root <none> and password file <none>
20:04:31.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.484058  7772 fs_manager.cc:362] Metadata directory not provided
20:04:31.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.484328  7772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:31.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.494053  7772 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.000s
20:04:31.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.499795  7795 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:31.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.501026  7772 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
20:04:31.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.501426  7772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:31.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "993f936550594542941a497816c4b19d"
20:04:31.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:58 on dist-test-slave-vt9x"
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.503721  7772 fs_report.cc:389] FS layout report
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/data
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:31.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:31.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:31.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:31.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:31.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.596148  7772 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:31.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.598503  7772 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:31.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.599123  7772 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:31.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.603122  7772 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:31.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.612076  7802 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:31.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.622514  7772 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:31.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.622874  7772 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.001s	sys 0.001s
20:04:31.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.623208  7772 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:31.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.631058  7772 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:31.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.631343  7772 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.005s	sys 0.000s
20:04:31.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.632018  7802 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap starting.
20:04:31.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.715413  7802 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Log is configured to *not* fsync() on all Append() calls
20:04:31.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.804842  7802 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:31.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.806679  7802 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap complete.
20:04:31.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.809401  7802 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent bootstrapping tablet: real 0.178s	user 0.136s	sys 0.032s
20:04:31.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.830861  7802 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:31.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.833467  7802 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993f936550594542941a497816c4b19d, State: Initialized, Role: FOLLOWER
20:04:31.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.834638  7802 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:31.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.840029  7802 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent starting tablet: real 0.030s	user 0.027s	sys 0.004s
20:04:31.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.869796  7772 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.65:40343
20:04:31.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.870136  7910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.65:40343 every 8 connection(s)
20:04:31.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.872784  7772 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:31.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.880450  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 7772
20:04:31.882 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:31.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.943823  7912 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:31.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.944754  7912 heartbeater.cc:461] Registering TS with master...
20:04:31.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.949055  7912 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:31.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.950328  7911 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:31.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.953720  7911 heartbeater.cc:461] Registering TS with master...
20:04:31.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.955423  7911 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:31.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.961287  6398 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:31.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.962936  7913 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:31.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.963523  7913 heartbeater.cc:461] Registering TS with master...
20:04:31.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.965396  7913 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:31.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.966696  6323 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:31.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.970158  6398 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.65:57383
20:04:31.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.971617  6245 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:31.975 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:31.976 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.66:36169
20:04:31.977 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

20:04:31.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:31.977826  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 7394
20:04:32.008 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:32.009 [DEBUG - kudu-nio-7] (Connection.java:239) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] handling channelInactive
20:04:32.009 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state READY due to: connection closed
20:04:32.025 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:32.026 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.027 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.027 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.048 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:32.048 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:32.049 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=3, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [18ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [18ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@995180485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@158bf645 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16a4eb5c -> wakeup thread Time-limited test)) after lookup
20:04:32.050 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:32.051 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.051 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.051 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.068 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:32.068 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:32.069 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=5, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=60), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [18ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [18ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [42ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [42ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [59ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@995180485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@158bf645 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16a4eb5c -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226))) after lookup
20:04:32.070 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:32.071 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.071 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.071 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.128 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:32.129 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:32.129 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=7, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=120), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [18ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [18ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [42ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [42ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [59ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [61ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [62ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [63ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [119ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@995180485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@158bf645 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16a4eb5c -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509))) after lookup
20:04:32.130 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:32.132 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.132 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.132 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.188 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:32.188 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:32.189 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=9, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=180), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [18ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [18ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [42ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [42ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [59ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [61ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [62ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [63ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [119ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [121ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [123ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [123ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [174ms] refreshing cache from master, [174ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [178ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@995180485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@158bf645 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509) -> (continuation of Deferred@287443457 after retry RPC@601143300), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16a4eb5c -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509) -> (continuation of Deferred@287443457 after retry RPC@601143300))) after lookup
20:04:32.189 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:32.191 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.191 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.191 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.868 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:32.868 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:32.869 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=11, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=860), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [18ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [18ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [42ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [42ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [59ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [61ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [62ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [63ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [119ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [121ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [123ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [123ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [174ms] refreshing cache from master, [174ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [178ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [180ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [182ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [182ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [854ms] refreshing cache from master, [854ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [858ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@995180485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@158bf645 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509) -> (continuation of Deferred@287443457 after retry RPC@601143300) -> (continuation of Deferred@1697757431 after retry RPC@1663248589), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16a4eb5c -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509) -> (continuation of Deferred@287443457 after retry RPC@601143300) -> (continuation of Deferred@1697757431 after retry RPC@1663248589))) after lookup
20:04:32.870 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:32.872 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.872 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.872 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:32.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:32.978024  7911 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:33.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.244884  7923 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:04:33.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.245553  7923 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:33.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.248313  7923 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:33.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:33.253427  7798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:33.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.260586  7680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a" is_pre_election: true
20:04:33.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:33.261300  7798 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:33.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.262379  7796 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 993f936550594542941a497816c4b19d; no voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482
20:04:33.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.263108  7923 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
20:04:33.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.455641  7928 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e35e6d7e70184db4b58023c7bcd30482)
20:04:33.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.456153  7928 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:33.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.457883  7928 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:33.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:33.460805  7613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:33.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:33.464926  7613 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:33.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.472301  7865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d" is_pre_election: true
20:04:33.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.473078  7865 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 5.
20:04:33.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.474519  7612 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d; no voters: e35e6d7e70184db4b58023c7bcd30482
20:04:33.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.475289  7928 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 6 FOLLOWER]: Leader pre-election won for term 7
20:04:33.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.475561  7928 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 6 FOLLOWER]: Starting leader election (detected failure of leader e35e6d7e70184db4b58023c7bcd30482)
20:04:33.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.475867  7928 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 6 FOLLOWER]: Advancing to term 7
20:04:33.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.483436  7928 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:33.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.485932  7928 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 7 election: Requested vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:33.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.487991  7865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d"
20:04:33.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.488704  7865 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 5 FOLLOWER]: Advancing to term 7
20:04:33.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:33.490916  7613 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.501042  7865 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 7.
20:04:33.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.502676  7612 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d; no voters: e35e6d7e70184db4b58023c7bcd30482
20:04:33.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.503767  7928 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Leader election won for term 7
20:04:33.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.505214  7928 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 LEADER]: Becoming Leader. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Running, Role: LEADER
20:04:33.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.506672  7928 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:33.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.520673  6398 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a reported cstate change: term changed from 6 to 7, leader changed from e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66) to 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67). New cstate: current_term: 7 leader_uuid: "31b9b4da6f9945578c783b9a7a597b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: UNKNOWN } } }
20:04:33.908 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:33.908 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:33.909 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=13, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1900), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [18ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [18ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [42ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [42ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [59ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [61ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [62ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [63ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [119ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [121ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [123ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [123ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [174ms] refreshing cache from master, [174ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [178ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [180ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [182ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [182ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [854ms] refreshing cache from master, [854ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [858ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [861ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [863ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [863ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [1894ms] refreshing cache from master, [1894ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1899ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@995180485(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@158bf645 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509) -> (continuation of Deferred@287443457 after retry RPC@601143300) -> (continuation of Deferred@1697757431 after retry RPC@1663248589) -> (continuation of Deferred@1736075978 after retry RPC@501568424), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16a4eb5c -> wakeup thread Time-limited test -> (continuation of Deferred@1474504551 after retry RPC@1421085226) -> (continuation of Deferred@1421076787 after retry RPC@547511509) -> (continuation of Deferred@287443457 after retry RPC@601143300) -> (continuation of Deferred@1697757431 after retry RPC@1663248589) -> (continuation of Deferred@1736075978 after retry RPC@501568424))) after lookup
20:04:33.909 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:33.911 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] Successfully connected to peer
20:04:33.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:33.923589  7613 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a -> Peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Couldn't send request to peer e35e6d7e70184db4b58023c7bcd30482. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:33.947 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x885546be, L:/127.0.0.1:45506 - R:/127.6.14.67:33469] using TOKEN/null
20:04:33.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.959610  7865 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 7 FOLLOWER]: Refusing update from remote peer 31b9b4da6f9945578c783b9a7a597b5a: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 26. (index mismatch)
20:04:33.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.961372  7928 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
20:04:33.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:33.977135  7949 mvcc.cc:204] Tried to move back new op lower bound from 7287335604489916416 to 7287335600657129472. Current Snapshot: MvccSnapshot[applied={T|T < 7287335604489916416}]
20:04:34.026 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.66:36169
20:04:34.026 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:34.027992  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:34.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:36169
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41631
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:34.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:34.413219  7956 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:34.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:34.414155  7956 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:34.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:34.415055  7956 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:34.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:34.453028  7956 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:34.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:34.454111  7956 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.66
20:04:34.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:34.505160  7956 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:34.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:36169
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41631
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:34.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:34.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:34.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:34.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:34.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:34.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:34.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:34.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:34.507328  7956 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:34.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:34.509426  7956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:34.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:34.528565  7964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:35.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:34.531351  7965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:35.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:35.949424  7967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:35.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:35.952705  7966 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1417 milliseconds
20:04:35.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.952806  7956 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.954351  7956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.957088  7956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:35.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.958598  7956 hybrid_clock.cc:648] HybridClock initialized: now 1779134675958539 us; error 69 us; skew 500 ppm
20:04:35.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.959525  7956 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:35.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.967785  7956 webserver.cc:533] Webserver started at http://127.6.14.66:41631/ using document root <none> and password file <none>
20:04:35.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.968986  7956 fs_manager.cc:362] Metadata directory not provided
20:04:35.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.969208  7956 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:35.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.979483  7956 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.001s
20:04:35.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.985327  7977 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:35.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.986840  7956 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
20:04:35.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.987282  7956 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:35.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:01 on dist-test-slave-vt9x"
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:35.989777  7956 fs_report.cc:389] FS layout report
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/data
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:35.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:35.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:35.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:35.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:35.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:36.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.051868  7956 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:36.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.053897  7956 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:36.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.054522  7956 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:36.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.057963  7956 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:36.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.065444  7984 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:36.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.076191  7956 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:36.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.076506  7956 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.003s	sys 0.000s
20:04:36.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.076756  7956 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:36.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.082834  7956 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:36.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.083083  7956 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
20:04:36.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.083848  7984 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap starting.
20:04:36.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.160065  7984 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Log is configured to *not* fsync() on all Append() calls
20:04:36.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.275269  7984 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:36.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.276623  7984 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap complete.
20:04:36.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.279035  7984 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent bootstrapping tablet: real 0.196s	user 0.148s	sys 0.041s
20:04:36.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.300232  7956 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.66:36169
20:04:36.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.300315  8091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.66:36169 every 8 connection(s)
20:04:36.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.304512  7956 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:36.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.305509  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 7956
20:04:36.307 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:36.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.312772  7984 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:36.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.317698  7984 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35e6d7e70184db4b58023c7bcd30482, State: Initialized, Role: FOLLOWER
20:04:36.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.319401  7984 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:36.360 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:36.364 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.67:33469
20:04:36.364 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:36.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.365468  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 7588
20:04:36.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.387019  7984 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent starting tablet: real 0.108s	user 0.045s	sys 0.018s
20:04:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.389725  8092 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.390353  8092 heartbeater.cc:461] Registering TS with master...
20:04:36.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.392093  8092 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:36.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.397693  6398 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:36.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.401190  6398 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.66:60401
20:04:36.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.406996  8092 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:36.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.415256  8094 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:36.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.415719  8094 heartbeater.cc:461] Registering TS with master...
20:04:36.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.416507  8094 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:36.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.418748  8093 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:36.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.419121  8093 heartbeater.cc:461] Registering TS with master...
20:04:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.419911  8093 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.420706  6245 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:36.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:36.424106  6323 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:36.427 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] handling channelInactive
20:04:36.427 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state READY due to: connection closed
20:04:36.428 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:36.429 [DEBUG - Time-limited test] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:36.430 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.430 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.431 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.448 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:36.448 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:36.448 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1163263260(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@721a9e0d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4f45e313 -> wakeup thread Time-limited test)) after lookup
20:04:36.449 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:36.450 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.450 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.451 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.467 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:36.468 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:36.468 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [21ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [22ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [23ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1163263260(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@721a9e0d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4f45e313 -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983))) after lookup
20:04:36.469 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:36.470 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.470 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.470 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.508 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:36.508 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:36.509 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [21ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [22ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [23ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [42ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [42ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [79ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1163263260(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@721a9e0d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4f45e313 -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120))) after lookup
20:04:36.510 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:36.511 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.511 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.511 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.628 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:36.628 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:36.628 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=200), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [21ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [22ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [23ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [42ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [42ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [79ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [81ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [83ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [83ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [195ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [199ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1163263260(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@721a9e0d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120) -> (continuation of Deferred@967190848 after retry RPC@271125158), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4f45e313 -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120) -> (continuation of Deferred@967190848 after retry RPC@271125158))) after lookup
20:04:36.629 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:36.630 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.631 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.631 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.788 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:36.789 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:36.789 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=360), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [21ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [22ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [23ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [42ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [42ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [79ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [81ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [83ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [83ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [195ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [199ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [201ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [203ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [203ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [355ms] refreshing cache from master, [355ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [360ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1163263260(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@721a9e0d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120) -> (continuation of Deferred@967190848 after retry RPC@271125158) -> (continuation of Deferred@1709494089 after retry RPC@1027638633), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4f45e313 -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120) -> (continuation of Deferred@967190848 after retry RPC@271125158) -> (continuation of Deferred@1709494089 after retry RPC@1027638633))) after lookup
20:04:36.790 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:36.791 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.791 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:36.792 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:37.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.476922  8105 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:37.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.477392  8105 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:37.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.479121  8105 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:37.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:37.486358  7796 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:37.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.494201  8047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:37.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.495172  8047 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 6.
20:04:37.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.496706  7798 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:37.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.497450  8105 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 7 FOLLOWER]: Leader pre-election won for term 8
20:04:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.497798  8105 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.498159  8105 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 7 FOLLOWER]: Advancing to term 8
20:04:37.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.503908  8105 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:37.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.505756  8105 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 8 election: Requested vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:37.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.507174  8047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:37.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.507879  8047 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 6 FOLLOWER]: Advancing to term 8
20:04:37.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:37.510725  7796 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:37.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.517271  8047 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 8.
20:04:37.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.518440  7798 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:37.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.519222  8105 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 8 FOLLOWER]: Leader election won for term 8
20:04:37.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.520543  8105 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 8 LEADER]: Becoming Leader. State: Replica: 993f936550594542941a497816c4b19d, State: Running, Role: LEADER
20:04:37.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.521574  8105 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:37.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.531124  6398 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d reported cstate change: term changed from 7 to 8, leader changed from 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67) to 993f936550594542941a497816c4b19d (127.6.14.65). New cstate: current_term: 8 leader_uuid: "993f936550594542941a497816c4b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: HEALTHY } } }
20:04:37.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.936965  8047 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 8 FOLLOWER]: Refusing update from remote peer 993f936550594542941a497816c4b19d: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
20:04:37.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:37.938695  8105 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
20:04:37.978 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:37.979 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:37.979 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1551), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [3ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [3ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [21ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [22ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [23ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [40ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [42ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [42ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [79ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [81ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [83ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [83ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [195ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [199ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [201ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [203ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [203ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [355ms] refreshing cache from master, [355ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [360ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [361ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [363ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [364ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [1535ms] refreshing cache from master, [1536ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1550ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1163263260(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@721a9e0d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120) -> (continuation of Deferred@967190848 after retry RPC@271125158) -> (continuation of Deferred@1709494089 after retry RPC@1027638633) -> (continuation of Deferred@903035751 after retry RPC@201427833), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4f45e313 -> wakeup thread Time-limited test -> (continuation of Deferred@488895811 after retry RPC@2107555983) -> (continuation of Deferred@1499327079 after retry RPC@221549120) -> (continuation of Deferred@967190848 after retry RPC@271125158) -> (continuation of Deferred@1709494089 after retry RPC@1027638633) -> (continuation of Deferred@903035751 after retry RPC@201427833))) after lookup
20:04:37.980 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:37.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:37.983654  7796 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d -> Peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Couldn't send request to peer 31b9b4da6f9945578c783b9a7a597b5a. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:37.993 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] Successfully connected to peer
20:04:38.024 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x5c34f555, L:/127.0.0.1:48748 - R:/127.6.14.65:40343] using TOKEN/null
20:04:38.075 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.67:33469
20:04:38.076 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:38.077659  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:38.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:38.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:38.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.431581  8131 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:38.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.432322  8131 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:38.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.432950  8131 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:38.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.472220  8131 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:38.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.473774  8131 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.67
20:04:38.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.474000  7796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111) [suppressed 3 similar messages]
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:38.516079  8131 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:38.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:38.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:38.517732  8131 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:38.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:38.519826  8131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:38.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.534818  8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:39.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:38.537511  8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:39.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:39.784119  8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:39.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:39.786917  8140 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:04:39.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.787029  8131 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:39.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.788623  8131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:39.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.791560  8131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:39.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.793107  8131 hybrid_clock.cc:648] HybridClock initialized: now 1779134679793060 us; error 49 us; skew 500 ppm
20:04:39.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.794166  8131 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:39.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.802712  8131 webserver.cc:533] Webserver started at http://127.6.14.67:45941/ using document root <none> and password file <none>
20:04:39.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.804096  8131 fs_manager.cc:362] Metadata directory not provided
20:04:39.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.804383  8131 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:39.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.813700  8131 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.004s	sys 0.000s
20:04:39.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.819211  8152 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:39.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.820644  8131 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
20:04:39.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.821105  8131 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:39.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:39.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:03 on dist-test-slave-vt9x"
20:04:39.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.824139  8131 fs_report.cc:389] FS layout report
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/data
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:39.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:39.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.913162  8131 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:39.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.915186  8131 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:39.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.915812  8131 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:39.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.919528  8131 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:39.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.928254  8159 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.942756  8131 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.943131  8131 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.017s	user 0.000s	sys 0.003s
20:04:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.943501  8131 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:39.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.949940  8131 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:39.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.950210  8131 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.005s	sys 0.000s
20:04:39.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:39.950907  8159 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap starting.
20:04:40.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.047422  8159 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Log is configured to *not* fsync() on all Append() calls
20:04:40.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.215370  8159 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:40.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.218092  8159 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap complete.
20:04:40.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.221258  8159 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent bootstrapping tablet: real 0.271s	user 0.176s	sys 0.070s
20:04:40.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.255630  8159 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:40.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.260641  8159 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Initialized, Role: FOLLOWER
20:04:40.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.262663  8159 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:40.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.264943  8131 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.67:33469
20:04:40.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.265844  8268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.67:33469 every 8 connection(s)
20:04:40.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.268967  8131 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:40.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.270790  8159 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent starting tablet: real 0.049s	user 0.043s	sys 0.001s
20:04:40.277 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.275132  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 8131
20:04:40.320 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:40.323 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.65:40343
20:04:40.323 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

20:04:40.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.325785  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 7772
20:04:40.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.358568  8270 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:40.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.359416  8270 heartbeater.cc:461] Registering TS with master...
20:04:40.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.365377  8270 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:40.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.371817  6397 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:40.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.376159  6397 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.67:46441
20:04:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.388685  8271 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.389117  8271 heartbeater.cc:461] Registering TS with master...
20:04:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.389892  8271 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.390676  8275 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.391021  8275 heartbeater.cc:461] Registering TS with master...
20:04:40.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.391644  8275 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:40.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.393959  6323 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:40.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:40.395025  6246 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:40.411 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] handling channelInactive
20:04:40.411 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state READY due to: connection closed
20:04:40.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:40.413 [DEBUG - Time-limited test] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:40.414 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.414 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.414 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.428 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:40.429 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:40.429 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@324526849(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@69201744 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@43543e7e -> wakeup thread Time-limited test)) after lookup
20:04:40.430 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:40.431 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.431 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.432 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.448 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:40.448 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:40.449 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [17ms] sending RPC to server 993f936550594542941a497816c4b19d, [19ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [20ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@324526849(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@69201744 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@43543e7e -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789))) after lookup
20:04:40.449 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:40.451 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.451 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.451 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.489 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:40.489 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:40.490 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [17ms] sending RPC to server 993f936550594542941a497816c4b19d, [19ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [20ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [37ms] sending RPC to server 993f936550594542941a497816c4b19d, [39ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [39ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [77ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@324526849(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@69201744 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@43543e7e -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640))) after lookup
20:04:40.491 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:40.492 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.492 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.492 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.629 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:40.629 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:40.630 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=217), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [17ms] sending RPC to server 993f936550594542941a497816c4b19d, [19ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [20ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [37ms] sending RPC to server 993f936550594542941a497816c4b19d, [39ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [39ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [77ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [78ms] sending RPC to server 993f936550594542941a497816c4b19d, [80ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [80ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [211ms] refreshing cache from master, [212ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [216ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@324526849(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@69201744 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640) -> (continuation of Deferred@1047232865 after retry RPC@1039813878), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@43543e7e -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640) -> (continuation of Deferred@1047232865 after retry RPC@1039813878))) after lookup
20:04:40.631 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:40.632 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.632 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:40.633 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:41.109 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:41.109 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:41.110 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=697), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [17ms] sending RPC to server 993f936550594542941a497816c4b19d, [19ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [20ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [37ms] sending RPC to server 993f936550594542941a497816c4b19d, [39ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [39ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [77ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [78ms] sending RPC to server 993f936550594542941a497816c4b19d, [80ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [80ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [211ms] refreshing cache from master, [212ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [216ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [218ms] sending RPC to server 993f936550594542941a497816c4b19d, [221ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [221ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [691ms] refreshing cache from master, [692ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [696ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@324526849(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@69201744 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640) -> (continuation of Deferred@1047232865 after retry RPC@1039813878) -> (continuation of Deferred@952454514 after retry RPC@1471626080), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@43543e7e -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640) -> (continuation of Deferred@1047232865 after retry RPC@1039813878) -> (continuation of Deferred@952454514 after retry RPC@1471626080))) after lookup
20:04:41.111 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:41.113 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:41.113 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state CONNECTING due to: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:41.113 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 993f936550594542941a497816c4b19d(127.6.14.65:40343) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343
20:04:41.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.383240  8270 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:41.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.489481  8281 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:04:41.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.490150  8281 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:41.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.493077  8281 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:41.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:41.498639  8154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:41.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:41.503299  8154 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:41.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.505482  8047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:41.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.507237  8155 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a; no voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482
20:04:41.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.508105  8281 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
20:04:41.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.516496  8286 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 993f936550594542941a497816c4b19d)
20:04:41.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.516985  8286 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:41.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.519734  8286 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:41.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:41.525664  7979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:41.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:41.530638  7979 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:41.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.535409  8224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a" is_pre_election: true
20:04:41.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.536453  8224 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35e6d7e70184db4b58023c7bcd30482 in term 7.
20:04:41.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.538166  7978 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482; no voters: 993f936550594542941a497816c4b19d
20:04:41.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.538964  8286 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 8 FOLLOWER]: Leader pre-election won for term 9
20:04:41.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.539278  8286 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 993f936550594542941a497816c4b19d)
20:04:41.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.539649  8286 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 8 FOLLOWER]: Advancing to term 9
20:04:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.545069  8286 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:41.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.546854  8286 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 9 election: Requested vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.547993  8224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:41.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.548538  8224 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 7 FOLLOWER]: Advancing to term 9
20:04:41.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:41.550575  7979 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111)
20:04:41.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.556550  8224 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35e6d7e70184db4b58023c7bcd30482 in term 9.
20:04:41.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.557556  7978 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, e35e6d7e70184db4b58023c7bcd30482; no voters: 993f936550594542941a497816c4b19d
20:04:41.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.558288  8286 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Leader election won for term 9
20:04:41.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.561264  8286 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 LEADER]: Becoming Leader. State: Replica: e35e6d7e70184db4b58023c7bcd30482, State: Running, Role: LEADER
20:04:41.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.563016  8286 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:41.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:41.576586  6397 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 reported cstate change: term changed from 8 to 9, leader changed from 993f936550594542941a497816c4b19d (127.6.14.65) to e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66). New cstate: current_term: 9 leader_uuid: "e35e6d7e70184db4b58023c7bcd30482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: UNKNOWN } } }
20:04:42.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:42.007642  7979 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 -> Peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Couldn't send request to peer 993f936550594542941a497816c4b19d. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:42.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:42.101187  8224 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 9 FOLLOWER]: Refusing update from remote peer e35e6d7e70184db4b58023c7bcd30482: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
20:04:42.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:42.102937  8286 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
20:04:42.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:42.988 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:42.989 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2576), Traces: [0ms] sending RPC to server 993f936550594542941a497816c4b19d, [2ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [3ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [17ms] sending RPC to server 993f936550594542941a497816c4b19d, [19ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [20ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [37ms] sending RPC to server 993f936550594542941a497816c4b19d, [39ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [39ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [77ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [78ms] sending RPC to server 993f936550594542941a497816c4b19d, [80ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [80ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [211ms] refreshing cache from master, [212ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [216ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [218ms] sending RPC to server 993f936550594542941a497816c4b19d, [221ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [221ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [691ms] refreshing cache from master, [692ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [696ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [699ms] sending RPC to server 993f936550594542941a497816c4b19d, [701ms] received response from server 993f936550594542941a497816c4b19d: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [701ms] delaying RPC due to: Network error: Failed to connect to peer 993f936550594542941a497816c4b19d(127.6.14.65:40343): Connection refused: /127.6.14.65:40343, [2571ms] refreshing cache from master, [2572ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [2576ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@324526849(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@69201744 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640) -> (continuation of Deferred@1047232865 after retry RPC@1039813878) -> (continuation of Deferred@952454514 after retry RPC@1471626080) -> (continuation of Deferred@415615386 after retry RPC@1984400857), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@43543e7e -> wakeup thread Time-limited test -> (continuation of Deferred@2134965840 after retry RPC@199427789) -> (continuation of Deferred@280322431 after retry RPC@475800640) -> (continuation of Deferred@1047232865 after retry RPC@1039813878) -> (continuation of Deferred@952454514 after retry RPC@1471626080) -> (continuation of Deferred@415615386 after retry RPC@1984400857))) after lookup
20:04:42.990 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:42.991 [DEBUG - kudu-nio-4] (Connection.java:757) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] Successfully connected to peer
20:04:43.025 [DEBUG - kudu-nio-4] (Negotiator.java:919) Authenticated connection [id: 0xf68ab01b, L:/127.0.0.1:39350 - R:/127.6.14.66:36169] using TOKEN/null
20:04:43.079 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.65:40343
20:04:43.079 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

20:04:43.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:43.081352  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:40343
20:04:43.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:04:43.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:43.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44657
20:04:43.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:04:43.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:43.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:43.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:43.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:43.473531  8320 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:43.474126  8320 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:43.474617  8320 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:43.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:43.511804  8320 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:43.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:43.512859  8320 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.65
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:43.564518  8320 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.65:40343
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.65
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44657
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.65
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/logs
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:43.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:43.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:43.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:43.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:43.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:43.566919  8320 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:43.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:43.569315  8320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:43.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:43.592142  8329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:44.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:44.274956  7979 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 -> Peer 993f936550594542941a497816c4b19d (127.6.14.65:40343): Couldn't send request to peer 993f936550594542941a497816c4b19d. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.65:40343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
20:04:44.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:43.593304  8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:44.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:44.972746  8332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:44.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:44.975734  8331 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1378 milliseconds
20:04:44.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.975908  8320 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:44.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.977380  8320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:44.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.980465  8320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:44.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.981942  8320 hybrid_clock.cc:648] HybridClock initialized: now 1779134684981896 us; error 49 us; skew 500 ppm
20:04:44.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.982878  8320 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:44.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.990407  8320 webserver.cc:533] Webserver started at http://127.6.14.65:44657/ using document root <none> and password file <none>
20:04:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.991564  8320 fs_manager.cc:362] Metadata directory not provided
20:04:44.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:44.991866  8320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:45.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.000773  8320 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.008s	sys 0.000s
20:04:45.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.006131  8340 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:45.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.007561  8320 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.001s
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.008008  8320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "993f936550594542941a497816c4b19d"
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:03:58 on dist-test-slave-vt9x"
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.010185  8320 fs_report.cc:389] FS layout report
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/wal
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/data
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:45.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:45.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:45.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.069345  8320 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:45.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.071105  8320 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:45.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.071571  8320 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:45.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.074954  8320 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:45.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.081569  8347 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:45.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.092597  8320 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:45.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.092857  8320 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.002s	sys 0.000s
20:04:45.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.093089  8320 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:45.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.098305  8320 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:45.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.098615  8320 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.003s	sys 0.000s
20:04:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.099067  8347 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap starting.
20:04:45.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.179944  8347 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Log is configured to *not* fsync() on all Append() calls
20:04:45.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.310832  8347 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:45.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.312435  8347 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Bootstrap complete.
20:04:45.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.315459  8347 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent bootstrapping tablet: real 0.217s	user 0.179s	sys 0.025s
20:04:45.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.338136  8347 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:45.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.340850  8347 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993f936550594542941a497816c4b19d, State: Initialized, Role: FOLLOWER
20:04:45.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.341995  8347 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:45.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.352115  8347 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d: Time spent starting tablet: real 0.036s	user 0.027s	sys 0.007s
20:04:45.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.389652  8320 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.65:40343
20:04:45.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.390555  8455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.65:40343 every 8 connection(s)
20:04:45.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.394285  8320 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-0/data/info.pb
20:04:45.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.397488  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 8320
20:04:45.399 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:45.472 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:45.473 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.66:36169
20:04:45.474 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

20:04:45.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.474601  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 7956
20:04:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.479511  8456 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.480378  8456 heartbeater.cc:461] Registering TS with master...
20:04:45.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.483469  8456 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:45.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.486248  8457 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:45.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.486642  8457 heartbeater.cc:461] Registering TS with master...
20:04:45.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.487715  8457 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:45.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.490293  8460 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:45.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.490653  8460 heartbeater.cc:461] Registering TS with master...
20:04:45.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.491181  6398 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:45.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.491636  8460 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:45.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.494282  6322 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:45.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.495065  6398 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.65:51879
20:04:45.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:45.498086  6245 ts_manager.cc:194] Re-registered known tserver with Master: 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:45.517 [DEBUG - kudu-nio-4] (Connection.java:239) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] handling channelInactive
20:04:45.517 [DEBUG - kudu-nio-4] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state READY due to: connection closed
20:04:45.518 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:45.518 [DEBUG - Time-limited test] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:45.520 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.520 [DEBUG - kudu-nio-5] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.520 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.528 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:45.529 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:45.529 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [2ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1475071220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@1d88aa95 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@377b2df3 -> wakeup thread Time-limited test)) after lookup
20:04:45.530 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:45.531 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.531 [DEBUG - kudu-nio-6] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.531 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.548 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:45.548 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:45.548 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [2ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [13ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [13ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1475071220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@1d88aa95 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@377b2df3 -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477))) after lookup
20:04:45.549 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:45.550 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.550 [DEBUG - kudu-nio-7] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.551 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.588 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:45.589 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:45.589 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [2ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [13ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [13ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [30ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [33ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [33ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [70ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1475071220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@1d88aa95 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@377b2df3 -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180))) after lookup
20:04:45.590 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:45.591 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.592 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.592 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.649 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:45.649 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:45.650 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=131), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [2ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [13ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [13ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [30ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [33ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [33ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [70ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [72ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [74ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [74ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [130ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1475071220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@1d88aa95 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@377b2df3 -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566))) after lookup
20:04:45.652 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:45.653 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.653 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.653 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.869 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:45.869 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:45.870 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=351), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [2ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [13ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [13ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [30ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [33ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [33ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [70ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [72ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [74ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [74ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [130ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [133ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [135ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [135ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [345ms] refreshing cache from master, [346ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [350ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1475071220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@1d88aa95 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566) -> (continuation of Deferred@1113569888 after retry RPC@2123052052), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@377b2df3 -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566) -> (continuation of Deferred@1113569888 after retry RPC@2123052052))) after lookup
20:04:45.871 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:45.872 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.873 [DEBUG - kudu-nio-2] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:45.873 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:46.408 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:46.408 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)[L]]}]
20:04:46.409 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=890), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [2ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [13ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [13ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [30ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [33ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [33ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [70ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [72ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [74ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [74ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [130ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [133ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [135ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [135ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [345ms] refreshing cache from master, [346ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [350ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [352ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [355ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [355ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [885ms] refreshing cache from master, [885ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [890ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1475071220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@1d88aa95 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566) -> (continuation of Deferred@1113569888 after retry RPC@2123052052) -> (continuation of Deferred@2032857892 after retry RPC@1587958657), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@377b2df3 -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566) -> (continuation of Deferred@1113569888 after retry RPC@2123052052) -> (continuation of Deferred@2032857892 after retry RPC@1587958657))) after lookup
20:04:46.410 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] connecting to peer
20:04:46.411 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:46.411 [DEBUG - kudu-nio-3] (Connection.java:709) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] cleaning up while in state CONNECTING due to: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:46.412 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169
20:04:46.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.502270  8456 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:46.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.522257  8467 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e35e6d7e70184db4b58023c7bcd30482)
20:04:46.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.522825  8467 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:46.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.524709  8467 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:46.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:46.527508  8155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:46.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:46.531260  8155 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:46.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.539232  8410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d" is_pre_election: true
20:04:46.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.540138  8410 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 8.
20:04:46.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.541703  8154 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d; no voters: e35e6d7e70184db4b58023c7bcd30482
20:04:46.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.542618  8467 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 9 FOLLOWER]: Leader pre-election won for term 10
20:04:46.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.542963  8467 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 9 FOLLOWER]: Starting leader election (detected failure of leader e35e6d7e70184db4b58023c7bcd30482)
20:04:46.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.543188  8467 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 9 FOLLOWER]: Advancing to term 10
20:04:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.548698  8467 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:46.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.550309  8467 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 10 election: Requested vote from peers e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:46.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.551651  8410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "31b9b4da6f9945578c783b9a7a597b5a" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d"
20:04:46.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.552331  8410 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 8 FOLLOWER]: Advancing to term 10
20:04:46.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:46.554122  8155 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111)
20:04:46.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.559154  8410 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b9b4da6f9945578c783b9a7a597b5a in term 10.
20:04:46.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.560681  8154 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d; no voters: e35e6d7e70184db4b58023c7bcd30482
20:04:46.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.561932  8467 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 10 FOLLOWER]: Leader election won for term 10
20:04:46.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.563519  8467 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 10 LEADER]: Becoming Leader. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Running, Role: LEADER
20:04:46.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.564628  8467 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:46.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.574344  6398 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a reported cstate change: term changed from 9 to 10, leader changed from e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66) to 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67). New cstate: current_term: 10 leader_uuid: "31b9b4da6f9945578c783b9a7a597b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: UNKNOWN } } }
20:04:46.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.968130  8410 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 10 FOLLOWER]: Refusing update from remote peer 31b9b4da6f9945578c783b9a7a597b5a: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
20:04:46.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:46.969885  8467 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
20:04:47.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.006965  8155 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a -> Peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Couldn't send request to peer e35e6d7e70184db4b58023c7bcd30482. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:47.287 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:47.287 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:47.288 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1769), Traces: [0ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [2ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [2ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [11ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [13ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [13ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [29ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [30ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [33ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [33ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [70ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [72ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [74ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [74ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [126ms] refreshing cache from master, [126ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [130ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [133ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [135ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [135ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [345ms] refreshing cache from master, [346ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [350ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [352ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [355ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [355ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [885ms] refreshing cache from master, [885ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [890ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [891ms] sending RPC to server e35e6d7e70184db4b58023c7bcd30482, [894ms] received response from server e35e6d7e70184db4b58023c7bcd30482: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [894ms] delaying RPC due to: Network error: Failed to connect to peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169): Connection refused: /127.6.14.66:36169, [1765ms] refreshing cache from master, [1765ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1769ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1475071220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@1d88aa95 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566) -> (continuation of Deferred@1113569888 after retry RPC@2123052052) -> (continuation of Deferred@2032857892 after retry RPC@1587958657) -> (continuation of Deferred@116262854 after retry RPC@1790469825), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@377b2df3 -> wakeup thread Time-limited test -> (continuation of Deferred@636484166 after retry RPC@1268076477) -> (continuation of Deferred@924915218 after retry RPC@1453875180) -> (continuation of Deferred@1845613541 after retry RPC@2051192566) -> (continuation of Deferred@1113569888 after retry RPC@2123052052) -> (continuation of Deferred@2032857892 after retry RPC@1587958657) -> (continuation of Deferred@116262854 after retry RPC@1790469825))) after lookup
20:04:47.288 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:47.290 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] Successfully connected to peer
20:04:47.309 [DEBUG - kudu-nio-4] (Negotiator.java:919) Authenticated connection [id: 0x996eb3d0, L:/127.0.0.1:55284 - R:/127.6.14.67:33469] using TOKEN/null
20:04:47.359 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.66:36169
20:04:47.359 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

20:04:47.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:47.360854  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:47.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:47.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:47.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:04:47.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:47.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:36169
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41631
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:47.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.734437  8494 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.735041  8494 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.735518  8494 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:47.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.774351  8494 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.775302  8494 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.66
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:47.818197  8494 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:47.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.66:36169
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.66
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41631
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.66
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/logs
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:47.819943  8494 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:47.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:47.821995  8494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:47.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.838852  8500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:49.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:49.286564  8155 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a -> Peer e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169): Couldn't send request to peer e35e6d7e70184db4b58023c7bcd30482. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.66:36169: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
20:04:49.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:47.839268  8501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:49.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:49.242519  8499 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 8494
20:04:49.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:49.404485  8494 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.566s	user 0.577s	sys 0.920s
20:04:49.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:49.406641  8494 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.568s	user 0.578s	sys 0.920s
20:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:49.407851  8503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:49.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:49.412177  8502 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1567 milliseconds
20:04:49.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.412250  8494 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:49.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.413683  8494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:49.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.416342  8494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:49.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.417826  8494 hybrid_clock.cc:648] HybridClock initialized: now 1779134689417678 us; error 163 us; skew 500 ppm
20:04:49.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.418689  8494 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.425697  8494 webserver.cc:533] Webserver started at http://127.6.14.66:41631/ using document root <none> and password file <none>
20:04:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.426718  8494 fs_manager.cc:362] Metadata directory not provided
20:04:49.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.426915  8494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:49.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.435580  8494 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
20:04:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.440816  8516 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.442132  8494 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
20:04:49.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.442438  8494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:49.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:49.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:01 on dist-test-slave-vt9x"
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.444620  8494 fs_report.cc:389] FS layout report
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/wal
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/data
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:49.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:49.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.507592  8494 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:49.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.509421  8494 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:49.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.509874  8494 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:49.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.512850  8494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:49.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.519387  8523 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:49.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.529372  8494 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:49.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.529649  8494 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.001s
20:04:49.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.529896  8494 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:49.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.535157  8494 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:49.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.535364  8494 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
20:04:49.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.536103  8523 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap starting.
20:04:49.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.632477  8523 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Log is configured to *not* fsync() on all Append() calls
20:04:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.757120  8523 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:49.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.758273  8523 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Bootstrap complete.
20:04:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.760190  8523 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent bootstrapping tablet: real 0.225s	user 0.164s	sys 0.055s
20:04:49.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.780297  8494 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.66:36169
20:04:49.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.780519  8630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.66:36169 every 8 connection(s)
20:04:49.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.780579  8523 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:49.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.783371  8523 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35e6d7e70184db4b58023c7bcd30482, State: Initialized, Role: FOLLOWER
20:04:49.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.784293  8494 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-1/data/info.pb
20:04:49.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.784420  8523 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:49.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.785132  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 8494
20:04:49.786 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:49.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.789275  8523 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482: Time spent starting tablet: real 0.029s	user 0.019s	sys 0.009s
20:04:49.848 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:49.849 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.14.67:33469
20:04:49.851 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:49.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.851845  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 8131
20:04:49.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.864319  8634 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:49.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.864952  8634 heartbeater.cc:461] Registering TS with master...
20:04:49.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.866544  8634 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:49.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.872215  8633 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:49.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.872618  8633 heartbeater.cc:461] Registering TS with master...
20:04:49.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.873703  8633 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:49.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.873920  6246 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:49.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.876716  8632 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:49.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.877135  8632 heartbeater.cc:461] Registering TS with master...
20:04:49.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.878199  8632 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:49.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.879963  6323 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:49.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.882438  6397 ts_manager.cc:194] Re-registered known tserver with Master: e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:49.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:49.886149  6397 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.66:36359
20:04:49.897 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] handling channelInactive
20:04:49.898 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state READY due to: connection closed
20:04:49.905 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:49.906 [DEBUG - Time-limited test] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:49.908 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.908 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.908 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.928 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:49.928 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:49.929 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1445225357(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@7f0389e6 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@730d6647 -> wakeup thread Time-limited test)) after lookup
20:04:49.930 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:49.931 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.931 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.931 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.948 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:49.949 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:49.949 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [23ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [25ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [25ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1445225357(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@7f0389e6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@730d6647 -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616))) after lookup
20:04:49.950 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:49.951 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.951 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.952 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:49.989 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:49.990 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [23ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [25ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [25ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [45ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1445225357(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@7f0389e6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@730d6647 -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671))) after lookup
20:04:49.991 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:49.992 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.992 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:49.992 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:50.028 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:50.028 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:50.029 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=122), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [23ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [25ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [25ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [45ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [84ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [86ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [86ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [122ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1445225357(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@7f0389e6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671) -> (continuation of Deferred@1566510871 after retry RPC@2130762601), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@730d6647 -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671) -> (continuation of Deferred@1566510871 after retry RPC@2130762601))) after lookup
20:04:50.029 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:50.031 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:50.031 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:50.031 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:50.188 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:50.189 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)[L],993f936550594542941a497816c4b19d(127.6.14.65:40343),e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:50.189 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=283), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [23ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [25ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [25ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [45ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [84ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [86ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [86ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [122ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [123ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [125ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [126ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [277ms] refreshing cache from master, [277ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [282ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1445225357(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@7f0389e6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671) -> (continuation of Deferred@1566510871 after retry RPC@2130762601) -> (continuation of Deferred@213408537 after retry RPC@1160269569), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@730d6647 -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671) -> (continuation of Deferred@1566510871 after retry RPC@2130762601) -> (continuation of Deferred@213408537 after retry RPC@1160269569))) after lookup
20:04:50.190 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] connecting to peer
20:04:50.191 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:50.192 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] cleaning up while in state CONNECTING due to: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:50.192 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469) for tablet 598f9144972d45d6bb2d77bd7f869093: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469
20:04:50.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:50.891567  8632 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:51.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.174566  8645 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:04:51.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.175158  8645 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:51.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.178068  8645 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), 993f936550594542941a497816c4b19d (127.6.14.65:40343)
20:04:51.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:51.183455  8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:51.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:51.188174  8517 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:51.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.191988  8410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "e35e6d7e70184db4b58023c7bcd30482" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "993f936550594542941a497816c4b19d" is_pre_election: true
20:04:51.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.193773  8518 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a, 993f936550594542941a497816c4b19d
20:04:51.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.194552  8645 raft_consensus.cc:2749] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
20:04:51.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.303567  8650 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:51.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.304176  8650 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:51.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.307026  8650 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:51.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:51.314181  8341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:51.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:51.319918  8341 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:51.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.326707  8586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482" is_pre_election: true
20:04:51.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.327495  8586 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 9.
20:04:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.329296  8343 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.330377  8650 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 10 FOLLOWER]: Leader pre-election won for term 11
20:04:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.330668  8650 raft_consensus.cc:493] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 31b9b4da6f9945578c783b9a7a597b5a)
20:04:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.330909  8650 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 10 FOLLOWER]: Advancing to term 11
20:04:51.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.336403  8650 raft_consensus.cc:515] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:51.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.338145  8650 leader_election.cc:290] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 11 election: Requested vote from peers 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469), e35e6d7e70184db4b58023c7bcd30482 (127.6.14.66:36169)
20:04:51.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.339603  8586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598f9144972d45d6bb2d77bd7f869093" candidate_uuid: "993f936550594542941a497816c4b19d" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "e35e6d7e70184db4b58023c7bcd30482"
20:04:51.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.340314  8586 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 9 FOLLOWER]: Advancing to term 11
20:04:51.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:51.342636  8341 leader_election.cc:336] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111)
20:04:51.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.348969  8586 raft_consensus.cc:2468] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993f936550594542941a497816c4b19d in term 11.
20:04:51.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.350208  8343 leader_election.cc:304] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 993f936550594542941a497816c4b19d, e35e6d7e70184db4b58023c7bcd30482; no voters: 31b9b4da6f9945578c783b9a7a597b5a
20:04:51.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.351015  8650 raft_consensus.cc:2804] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 11 FOLLOWER]: Leader election won for term 11
20:04:51.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.353292  8650 raft_consensus.cc:697] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [term 11 LEADER]: Becoming Leader. State: Replica: 993f936550594542941a497816c4b19d, State: Running, Role: LEADER
20:04:51.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.354628  8650 consensus_queue.cc:237] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:51.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.365120  6397 catalog_manager.cc:5696] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d reported cstate change: term changed from 10 to 11, leader changed from 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67) to 993f936550594542941a497816c4b19d (127.6.14.65). New cstate: current_term: 11 leader_uuid: "993f936550594542941a497816c4b19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } health_report { overall_health: HEALTHY } } }
20:04:51.608 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 598f9144972d45d6bb2d77bd7f869093 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928' with partition [<start>, <end>)
20:04:51.609 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=598f9144972d45d6bb2d77bd7f869093@[31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469),993f936550594542941a497816c4b19d(127.6.14.65:40343)[L],e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)]}]
20:04:51.609 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=598f9144972d45d6bb2d77bd7f869093, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1703), Traces: [0ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [2ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [2ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [22ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [23ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [25ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [25ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [42ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [44ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [45ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [46ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [77ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [82ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [84ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [86ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [86ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [122ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [123ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [125ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [126ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [277ms] refreshing cache from master, [277ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [282ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, [284ms] sending RPC to server 31b9b4da6f9945578c783b9a7a597b5a, [286ms] received response from server 31b9b4da6f9945578c783b9a7a597b5a: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [286ms] delaying RPC due to: Network error: Failed to connect to peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469): Connection refused: /127.6.14.67:33469, [1697ms] refreshing cache from master, [1697ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.14.124:44305, [1702ms] Sub RPC GetTableLocations: received response from server master-127.6.14.124:44305: OK, deferred=Deferred@1445225357(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$238/0x00007fb75c336bb8@7f0389e6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671) -> (continuation of Deferred@1566510871 after retry RPC@2130762601) -> (continuation of Deferred@213408537 after retry RPC@1160269569) -> (continuation of Deferred@1614448524 after retry RPC@1735236813), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@730d6647 -> wakeup thread Time-limited test -> (continuation of Deferred@1364902362 after retry RPC@1232976616) -> (continuation of Deferred@1386533041 after retry RPC@1982503671) -> (continuation of Deferred@1566510871 after retry RPC@2130762601) -> (continuation of Deferred@213408537 after retry RPC@1160269569) -> (continuation of Deferred@1614448524 after retry RPC@1735236813))) after lookup
20:04:51.610 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] connecting to peer
20:04:51.612 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] Successfully connected to peer
20:04:51.634 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xbe857a4b, L:/127.0.0.1:39256 - R:/127.6.14.65:40343] using TOKEN/null
20:04:51.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.644994  8586 raft_consensus.cc:1275] T 598f9144972d45d6bb2d77bd7f869093 P e35e6d7e70184db4b58023c7bcd30482 [term 11 FOLLOWER]: Refusing update from remote peer 993f936550594542941a497816c4b19d: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 42. (index mismatch)
20:04:51.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.647004  8655 consensus_queue.cc:1048] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.001s
20:04:51.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:51.648066  8341 consensus_peers.cc:597] T 598f9144972d45d6bb2d77bd7f869093 P 993f936550594542941a497816c4b19d -> Peer 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469): Couldn't send request to peer 31b9b4da6f9945578c783b9a7a597b5a. Status: Network error: Client connection negotiation failed: client connection to 127.6.14.67:33469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:04:51.725 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.14.67:33469
20:04:51.725 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

20:04:51.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:51.727406  6201 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:04:52.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:52.072872  8675 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
20:04:52.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:52.073455  8675 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:04:52.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:52.073946  8675 flags.cc:432] Enabled unsafe flag: --never_fsync=true
20:04:52.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:52.110986  8675 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
20:04:52.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:52.112033  8675 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.14.67
20:04:52.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:52.155177  8675 tablet_server_runner.cc:78] Tablet server non-default flags:
20:04:52.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:04:52.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.14.84:33431
20:04:52.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:04:52.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:04:52.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.14.67:33469
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.14.67
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45941
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.14.126:43963,127.6.14.125:41525,127.6.14.124:44305
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.14.67
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/logs
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 19:43:23 UTC on e7f111948823
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11963
20:04:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:04:52.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:52.156823  8675 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:04:52.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:52.158696  8675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
20:04:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:52.174906  8681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:53.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:52.175145  8682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:53.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:53.535513  8684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:04:53.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:53.538162  8683 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1360 milliseconds
20:04:53.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.538282  8675 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:04:53.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.539719  8675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:04:53.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.542222  8675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:04:53.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.543609  8675 hybrid_clock.cc:648] HybridClock initialized: now 1779134693543578 us; error 48 us; skew 500 ppm
20:04:53.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.544656  8675 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:04:53.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.552014  8675 webserver.cc:533] Webserver started at http://127.6.14.67:45941/ using document root <none> and password file <none>
20:04:53.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.553067  8675 fs_manager.cc:362] Metadata directory not provided
20:04:53.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.553267  8675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:04:53.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.562660  8675 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.000s
20:04:53.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.569546  8693 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
20:04:53.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.571079  8675 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.003s	sys 0.002s
20:04:53.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.571517  8675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data,/tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:53.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31b9b4da6f9945578c783b9a7a597b5a"
20:04:53.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 20:04:03 on dist-test-slave-vt9x"
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.573791  8675 fs_report.cc:389] FS layout report
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/wal
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/data
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:04:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:04:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:04:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:04:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:04:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:04:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:04:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:04:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:04:53.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.636572  8675 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
20:04:53.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.639182  8675 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:04:53.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.639900  8675 kserver.cc:163] Server-wide thread pool size limit: 3276
20:04:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.644201  8675 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
20:04:53.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.654165  8700 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
20:04:53.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.668536  8675 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
20:04:53.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.668967  8675 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.017s	user 0.002s	sys 0.000s
20:04:53.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.669371  8675 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
20:04:53.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.680028  8675 ts_tablet_manager.cc:616] Registered 1 tablets
20:04:53.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.680392  8675 ts_tablet_manager.cc:595] Time spent register tablets: real 0.011s	user 0.007s	sys 0.005s
20:04:53.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.681072  8700 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap starting.
20:04:53.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.780251  8700 log.cc:826] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Log is configured to *not* fsync() on all Append() calls
20:04:53.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.919945  8700 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:04:53.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.921177  8700 tablet_bootstrap.cc:492] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Bootstrap complete.
20:04:53.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.922662  8700 ts_tablet_manager.cc:1403] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent bootstrapping tablet: real 0.242s	user 0.164s	sys 0.061s
20:04:53.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.942780  8700 raft_consensus.cc:359] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:53.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.945446  8700 raft_consensus.cc:740] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b9b4da6f9945578c783b9a7a597b5a, State: Initialized, Role: FOLLOWER
20:04:53.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.946745  8700 consensus_queue.cc:260] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b9b4da6f9945578c783b9a7a597b5a" member_type: VOTER last_known_addr { host: "127.6.14.67" port: 33469 } } peers { permanent_uuid: "e35e6d7e70184db4b58023c7bcd30482" member_type: VOTER last_known_addr { host: "127.6.14.66" port: 36169 } } peers { permanent_uuid: "993f936550594542941a497816c4b19d" member_type: VOTER last_known_addr { host: "127.6.14.65" port: 40343 } }
20:04:53.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.956537  8700 ts_tablet_manager.cc:1434] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a: Time spent starting tablet: real 0.034s	user 0.031s	sys 0.005s
20:04:53.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.991403  8675 rpc_server.cc:307] RPC server started. Bound to: 127.6.14.67:33469
20:04:53.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.992328  8808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.14.67:33469 every 8 connection(s)
20:04:53.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:53.994768  8675 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskwwkcsW/test-tmp/mini-kudu-cluster11914052779106923471/ts-2/data/info.pb
20:04:54.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.002835  6201 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu as pid 8675
20:04:54.003 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
20:04:54.047 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:54.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.071593  8809 heartbeater.cc:344] Connected to a master server at 127.6.14.124:44305
20:04:54.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.072408  8809 heartbeater.cc:461] Registering TS with master...
20:04:54.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.074900  8809 heartbeater.cc:507] Master 127.6.14.124:44305 requested a full tablet report, sending...
20:04:54.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.087324  6398 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:54.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.094902  6398 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.14.67:46017
20:04:54.097 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779134644928, tablet=598f9144972d45d6bb2d77bd7f869093, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
20:04:54.105 [DEBUG - main] (Connection.java:575) [peer master-127.6.14.125:41525(127.6.14.125:41525)] disconnecting while in state READY
20:04:54.106 [DEBUG - main] (Connection.java:575) [peer 31b9b4da6f9945578c783b9a7a597b5a(127.6.14.67:33469)] disconnecting while in state TERMINATED
20:04:54.106 [DEBUG - main] (Connection.java:575) [peer master-127.6.14.124:44305(127.6.14.124:44305)] disconnecting while in state READY
20:04:54.107 [DEBUG - main] (Connection.java:575) [peer e35e6d7e70184db4b58023c7bcd30482(127.6.14.66:36169)] disconnecting while in state TERMINATED
20:04:54.107 [DEBUG - main] (Connection.java:575) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] disconnecting while in state READY
20:04:54.111 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.6.14.125:41525(127.6.14.125:41525)] handling channelInactive
20:04:54.112 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.6.14.124:44305(127.6.14.124:44305)] handling channelInactive
20:04:54.112 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.6.14.124:44305(127.6.14.124:44305)] cleaning up while in state READY due to: connection closed
20:04:54.115 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.6.14.125:41525(127.6.14.125:41525)] cleaning up while in state READY due to: connection closed
20:04:54.118 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] handling channelInactive
20:04:54.118 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 993f936550594542941a497816c4b19d(127.6.14.65:40343)] cleaning up while in state READY due to: connection closed
20:04:54.119 [DEBUG - main] (Connection.java:575) [peer master-127.6.14.126:43963(127.6.14.126:43963)] disconnecting while in state READY
20:04:54.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.119392  8813 heartbeater.cc:344] Connected to a master server at 127.6.14.126:43963
20:04:54.120 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2704) Releasing all remaining resources
20:04:54.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.120162  8813 heartbeater.cc:461] Registering TS with master...
20:04:54.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.120853  8810 heartbeater.cc:344] Connected to a master server at 127.6.14.125:41525
20:04:54.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.121430  8810 heartbeater.cc:461] Registering TS with master...
20:04:54.122 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.6.14.126:43963(127.6.14.126:43963)] handling channelInactive
20:04:54.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.122588  8810 heartbeater.cc:507] Master 127.6.14.125:41525 requested a full tablet report, sending...
20:04:54.123 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.6.14.126:43963(127.6.14.126:43963)] cleaning up while in state READY due to: connection closed
20:04:54.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.123216  8763 raft_consensus.cc:3060] T 598f9144972d45d6bb2d77bd7f869093 P 31b9b4da6f9945578c783b9a7a597b5a [term 10 FOLLOWER]: Advancing to term 11
20:04:54.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.124342  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 8320
20:04:54.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.121438  8813 heartbeater.cc:507] Master 127.6.14.126:43963 requested a full tablet report, sending...
20:04:54.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.137876  8809 heartbeater.cc:499] Master 127.6.14.124:44305 was elected leader, sending a full tablet report...
20:04:54.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 20:04:54.168874  8695 connection.cc:582] server connection from 127.6.14.65:59555 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
20:04:54.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.171132  6323 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:54.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.173329  6245 ts_manager.cc:194] Re-registered known tserver with Master: 31b9b4da6f9945578c783b9a7a597b5a (127.6.14.67:33469)
20:04:54.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.177809  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 8494
20:04:54.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.214960  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 8675
20:04:54.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.249629  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 6213
20:04:54.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.284696  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 6283
20:04:54.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 20:04:54.321753  6201 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskwwkcsW/build/tsan/bin/kudu with pid 6360
20:04:55.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T20:04:55Z chronyd exiting

Time: 69.649

OK (1 test)

