JUnit version 4.13.2
.17:13:19.959 [INFO - main] (RandomUtils.java:49) Using random seed: 1783530799953
17:13:20.007 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
17:13:20.063 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
17:13:20.076 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskMwcjhq/test-tmp
17:13:20.078 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
17:13:20.082 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskMwcjhq/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
17:13:20.289 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

17:13:20.454 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
17:13:20.456 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

17:13:20.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.460624  6809 mini_oidc.cc:152] Starting JWKS server
17:13:20.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.464920  6809 webserver.cc:460] Webserver started at http://127.0.0.1:35497/ using document root <none> and password file <none>
17:13:20.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.465497  6809 mini_oidc.cc:189] Starting OIDC Discovery server
17:13:20.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.467644  6809 webserver.cc:460] Webserver started at http://127.0.0.1:35889/ using document root <none> and password file <none>
17:13:20.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T17:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
17:13:20.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T17:13:20Z Disabled control of system clock
17:13:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.515648  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/wal
17:13:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data
17:13:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:20.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:20.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:20.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:20.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:20.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:20.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/logs
17:13:20.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data/info.pb
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
17:13:20.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
17:13:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
17:13:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.126:39943
17:13:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.126
17:13:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:20.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
17:13:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
17:13:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35497/jwks.json with env {}
17:13:20.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.820300  6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:20.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.821189  6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:20.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.822024  6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:20.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.854467  6821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
17:13:20.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.854755  6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:20.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.855028  6821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
17:13:20.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.855223  6821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
17:13:20.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.855469  6821 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
17:13:20.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.855633  6821 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35497/jwks.json
17:13:20.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.892387  6821 master_runner.cc:386] Master server non-default flags:
17:13:20.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:20.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:20.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:20.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:20.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data
17:13:20.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/wal
17:13:20.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
17:13:20.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:20.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:20.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:20.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
17:13:20.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.126:39943
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35497/jwks.json
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data/info.pb
17:13:20.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.126
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/logs
17:13:20.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:20.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:20.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.893884  6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:20.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.896121  6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:20.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.916616  6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:22.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.916774  6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:22.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:20.917510  6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:22.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:20.918311  6821 server_base.cc:1034] running on GCE node
17:13:22.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.100633  6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:22.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.103138  6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:22.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.104575  6821 hybrid_clock.cc:648] HybridClock initialized: now 1783530802104546 us; error 39 us; skew 500 ppm
17:13:22.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.105500  6821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:22.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.116987  6821 webserver.cc:460] Webserver started at http://127.6.166.126:41475/ using document root <none> and password file <none>
17:13:22.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.118041  6821 fs_manager.cc:359] Metadata directory not provided
17:13:22.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.118269  6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:22.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.118799  6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
17:13:22.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.123678  6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data/instance:
17:13:22.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "efa10a1a0d7f4013a55cdf4770819773"
17:13:22.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:22 on dist-test-slave-v72b"
17:13:22.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.125108  6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/wal/instance:
17:13:22.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "efa10a1a0d7f4013a55cdf4770819773"
17:13:22.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:22 on dist-test-slave-v72b"
17:13:22.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.132215  6821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.003s
17:13:22.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.137934  6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:22.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.139037  6821 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
17:13:22.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.139413  6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/wal
17:13:22.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "efa10a1a0d7f4013a55cdf4770819773"
17:13:22.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:22 on dist-test-slave-v72b"
17:13:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.139820  6821 fs_report.cc:383] FS layout report
17:13:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/wal
17:13:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/wal
17:13:22.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data/data
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:22.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:22.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.177721  6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:22.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.179401  6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:22.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.179929  6821 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:22.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.249269  6821 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.126:39943
17:13:22.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.249352  6888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.126:39943 every 8 connection(s)
17:13:22.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.252144  6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/data/info.pb
17:13:22.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.257584  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 6821
17:13:22.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.257953  6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
17:13:22.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.258057  6809 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-0/wal/instance
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.261682  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/wal
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:22.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/logs
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data/info.pb
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:22.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.125:39419
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.125
17:13:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
17:13:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:22.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
17:13:22.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35497/jwks.json with env {}
17:13:22.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.275715  6889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } has no permanent_uuid. Determining permanent_uuid...
17:13:22.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.309154  6889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } has no permanent_uuid. Determining permanent_uuid...
17:13:22.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.311901  6838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:22.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.316438  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.125:39419: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:22.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.365993  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } attempt: 1
17:13:22.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.369894  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.125:39419: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:22.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.438316  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } attempt: 2
17:13:22.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.442337  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.125:39419: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:22.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.533741  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } attempt: 3
17:13:22.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.537823  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.125:39419: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:22.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.592221  6891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:22.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.592790  6891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:22.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.593364  6891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:22.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.624089  6891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
17:13:22.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.624377  6891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:22.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.624563  6891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
17:13:22.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.624739  6891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
17:13:22.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.624912  6891 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
17:13:22.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.625063  6891 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35497/jwks.json
17:13:22.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.659828  6891 master_runner.cc:386] Master server non-default flags:
17:13:22.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:22.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:22.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:22.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/wal
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.125:39419
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:22.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35497/jwks.json
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data/info.pb
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.125
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/logs
17:13:22.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:22.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:22.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:22.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.661031  6891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:22.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.662643  6891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:22.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.679219  6903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:22.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.681200  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } attempt: 4
17:13:22.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.684621  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.125:39419: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:22.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.984087  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } attempt: 5
17:13:22.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.988600  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.125:39419: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:23.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.535990  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } attempt: 6
17:13:23.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:23.543443  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.125:39419: Network error: Client connection negotiation failed: client connection to 127.6.166.125:39419: connect: Connection refused (error 111)
17:13:23.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:22.681161  6891 server_base.cc:1034] running on GCE node
17:13:23.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.679422  6901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:23.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:22.679338  6900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:23.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.901119  6891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:23.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.904196  6891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.905637  6891 hybrid_clock.cc:648] HybridClock initialized: now 1783530803905576 us; error 79 us; skew 500 ppm
17:13:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.906507  6891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:23.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.913378  6891 webserver.cc:460] Webserver started at http://127.6.166.125:37501/ using document root <none> and password file <none>
17:13:23.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.914486  6891 fs_manager.cc:359] Metadata directory not provided
17:13:23.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.914676  6891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:23.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.915059  6891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
17:13:23.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.920193  6891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data/instance:
17:13:23.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "09b11d4016dc4cdcab5ca01381b23b3c"
17:13:23.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:23 on dist-test-slave-v72b"
17:13:23.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.921315  6891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/wal/instance:
17:13:23.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "09b11d4016dc4cdcab5ca01381b23b3c"
17:13:23.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:23 on dist-test-slave-v72b"
17:13:23.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.928429  6891 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.001s
17:13:23.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.934026  6912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:23.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.935129  6891 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
17:13:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.935721  6891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/wal
17:13:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "09b11d4016dc4cdcab5ca01381b23b3c"
17:13:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:23 on dist-test-slave-v72b"
17:13:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.936038  6891 fs_report.cc:383] FS layout report
17:13:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/wal
17:13:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/wal
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data/data
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:23.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:23.999366  6891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:24.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.001003  6891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:24.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.001459  6891 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:24.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.075701  6891 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.125:39419
17:13:24.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.075749  6963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.125:39419 every 8 connection(s)
17:13:24.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.078758  6891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/data/info.pb
17:13:24.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.084076  6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
17:13:24.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.086237  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 6891
17:13:24.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.086577  6809 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-1/wal/instance
17:13:24.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.088585  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/wal
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/logs
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data/info.pb
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:24.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.124:44315
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.124
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:24.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:24.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:24.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:24.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
17:13:24.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:24.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
17:13:24.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35497/jwks.json with env {}
17:13:24.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.098774  6964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } has no permanent_uuid. Determining permanent_uuid...
17:13:24.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.122972  6964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } has no permanent_uuid. Determining permanent_uuid...
17:13:24.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.138223  6964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } has no permanent_uuid. Determining permanent_uuid...
17:13:24.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.140441  6914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.144182  6964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.193706  6964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 1
17:13:24.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.197886  6964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.266359  6964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 2
17:13:24.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.270149  6964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.361552  6964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 3
17:13:24.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.365934  6964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.394757  6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:24.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.395336  6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:24.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.395998  6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:24.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.426469  6966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
17:13:24.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.426751  6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:24.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.426932  6966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
17:13:24.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.427107  6966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
17:13:24.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.427353  6966 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
17:13:24.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.427516  6966 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35497/jwks.json
17:13:24.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.461917  6966 master_runner.cc:386] Master server non-default flags:
17:13:24.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:24.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:24.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:24.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:24.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data
17:13:24.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/wal
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.124:44315
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35497/jwks.json
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data/info.pb
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.124
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:24.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/logs
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:24.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.463192  6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:24.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.465317  6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:24.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.482591  6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:24.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.509302  6964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 4
17:13:24.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.513871  6964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.603847  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } attempt: 7
17:13:24.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.619288  6889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } has no permanent_uuid. Determining permanent_uuid...
17:13:24.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.624657  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.683305  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 1
17:13:24.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.688089  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.769544  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 2
17:13:24.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.774866  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.813292  6964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 5
17:13:24.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.817521  6964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:24.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.860278  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 3
17:13:24.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.864697  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:25.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.005089  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 4
17:13:25.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:25.008713  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:25.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.292112  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 5
17:13:25.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:25.296451  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:25.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.364928  6964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 6
17:13:25.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:25.372289  6964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:25.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.482614  6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:25.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:24.482990  6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:25.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:24.484827  6966 server_base.cc:1034] running on GCE node
17:13:25.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.712325  6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.714797  6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.716161  6966 hybrid_clock.cc:648] HybridClock initialized: now 1783530805716132 us; error 38 us; skew 500 ppm
17:13:25.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.717022  6966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:25.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.723182  6966 webserver.cc:460] Webserver started at http://127.6.166.124:40473/ using document root <none> and password file <none>
17:13:25.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.724273  6966 fs_manager.cc:359] Metadata directory not provided
17:13:25.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.724517  6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:25.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.725023  6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
17:13:25.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.730166  6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data/instance:
17:13:25.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f61cfe945c284afa8b5ddb9ed0c865b5"
17:13:25.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:25 on dist-test-slave-v72b"
17:13:25.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.731472  6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/wal/instance:
17:13:25.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f61cfe945c284afa8b5ddb9ed0c865b5"
17:13:25.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:25 on dist-test-slave-v72b"
17:13:25.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.738675  6966 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
17:13:25.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.744474  6990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:25.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.745558  6966 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
17:13:25.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.745900  6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/wal
17:13:25.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f61cfe945c284afa8b5ddb9ed0c865b5"
17:13:25.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:25 on dist-test-slave-v72b"
17:13:25.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.746289  6966 fs_report.cc:383] FS layout report
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/wal
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/wal
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data/data
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:25.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:25.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.784490  6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:25.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.786041  6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:25.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.786566  6966 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:25.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.848876  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 6
17:13:25.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:25.856216  6889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.166.124:44315: Network error: Client connection negotiation failed: client connection to 127.6.166.124:44315: connect: Connection refused (error 111)
17:13:25.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.858465  6966 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.124:44315
17:13:25.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.858590  7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.124:44315 every 8 connection(s)
17:13:25.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.861115  6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/data/info.pb
17:13:25.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.866276  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
17:13:25.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.870026  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 6966
17:13:25.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.870471  6809 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/master-2/wal/instance
17:13:25.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.881847  7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } has no permanent_uuid. Determining permanent_uuid...
17:13:25.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.915643  7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } has no permanent_uuid. Determining permanent_uuid...
17:13:25.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.928659  7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } has no permanent_uuid. Determining permanent_uuid...
17:13:25.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.944571  7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5: Bootstrap starting.
17:13:25.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.950291  7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5: Neither blocks nor log segments found. Creating new log.
17:13:25.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.952140  7044 log.cc:824] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5: Log is configured to *not* fsync() on all Append() calls
17:13:25.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.957312  7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5: No bootstrap required, opened a new log
17:13:25.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.977421  7044 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:25.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.978147  7044 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
17:13:25.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.978384  7044 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61cfe945c284afa8b5ddb9ed0c865b5, State: Initialized, Role: FOLLOWER
17:13:25.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.979107  7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:25.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.981571  7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:25.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.982424  7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: This master's current role is: FOLLOWER
17:13:25.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:25.982867  7044 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: configured and running, proceeding with master startup.
17:13:26.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:26.009181  7064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
17:13:26.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:26.009541  7064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
17:13:26.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.003599  6966 master_runner.cc:186] Error getting master registration for 127.6.166.126:39943: OK, instance_id { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" instance_seqno: 1783530802230269 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
17:13:26.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.027908  6966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
17:13:26.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:26.041857  6996 tablet.cc:2369] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
17:13:26.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.374465  7053 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:13:26.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.374974  7053 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:26.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.377125  7053 leader_election.cc:288] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efa10a1a0d7f4013a55cdf4770819773 (127.6.166.126:39943), 09b11d4016dc4cdcab5ca01381b23b3c (127.6.166.125:39419)
17:13:26.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.432693  6964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 7
17:13:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.454282  6964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: Bootstrap starting.
17:13:26.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.459726  6964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: Neither blocks nor log segments found. Creating new log.
17:13:26.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.464308  6964 log.cc:824] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: Log is configured to *not* fsync() on all Append() calls
17:13:26.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.469914  6964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: No bootstrap required, opened a new log
17:13:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.501945  6964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.502727  6964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
17:13:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.502992  6964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b11d4016dc4cdcab5ca01381b23b3c, State: Initialized, Role: FOLLOWER
17:13:26.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.503875  6964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:26.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.506508  7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:26.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.507591  7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [sys.catalog]: This master's current role is: FOLLOWER
17:13:26.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.508373  6964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [sys.catalog]: configured and running, proceeding with master startup.
17:13:26.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:26.508767  6918 tablet.cc:2369] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
17:13:26.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.508518  6939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" is_pre_election: true
17:13:26.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.509407  6939 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f61cfe945c284afa8b5ddb9ed0c865b5 in term 0.
17:13:26.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.510788  6991 leader_election.cc:302] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b11d4016dc4cdcab5ca01381b23b3c, f61cfe945c284afa8b5ddb9ed0c865b5; no voters: 
17:13:26.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.512046  7053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
17:13:26.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.512393  7053 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
17:13:26.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.512693  7053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 0 FOLLOWER]: Advancing to term 1
17:13:26.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.518814  7053 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:26.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.523815  6939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b11d4016dc4cdcab5ca01381b23b3c"
17:13:26.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.524416  6939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [term 0 FOLLOWER]: Advancing to term 1
17:13:26.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.530848  6939 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f61cfe945c284afa8b5ddb9ed0c865b5 in term 1.
17:13:26.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.523156  7053 leader_election.cc:288] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [CANDIDATE]: Term 1 election: Requested vote from peers efa10a1a0d7f4013a55cdf4770819773 (127.6.166.126:39943), 09b11d4016dc4cdcab5ca01381b23b3c (127.6.166.125:39419)
17:13:26.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.532049  6991 leader_election.cc:302] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b11d4016dc4cdcab5ca01381b23b3c, f61cfe945c284afa8b5ddb9ed0c865b5; no voters: 
17:13:26.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.532765  7053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 1 FOLLOWER]: Leader election won for term 1
17:13:26.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.539611  7053 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [term 1 LEADER]: Becoming Leader. State: Replica: f61cfe945c284afa8b5ddb9ed0c865b5, State: Running, Role: LEADER
17:13:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.541301  7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:26.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.563534  7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61cfe945c284afa8b5ddb9ed0c865b5. Latest consensus state: current_term: 1 leader_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:26.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.564867  7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: This master's current role is: LEADER
17:13:26.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.568089  7080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
17:13:26.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.575207  7080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
17:13:26.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.581635  7082 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
17:13:26.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.606343  6939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [term 1 FOLLOWER]: Refusing update from remote peer f61cfe945c284afa8b5ddb9ed0c865b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
17:13:26.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.608006  7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
17:13:26.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.585495  6891 master_runner.cc:186] Error getting master registration for 127.6.166.126:39943: OK, instance_id { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" instance_seqno: 1783530802230269 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
17:13:26.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.626812  6891 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
17:13:26.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.635648  7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:26.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.636700  7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: This master's current role is: LEADER
17:13:26.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.641440  7080 catalog_manager.cc:1347] Generated new cluster ID: 8118dd94be084fc8af6ea0af087d19cc
17:13:26.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.641726  7080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
17:13:26.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.649463  7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61cfe945c284afa8b5ddb9ed0c865b5. Latest consensus state: current_term: 1 leader_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:26.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.654912  7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [sys.catalog]: This master's current role is: FOLLOWER
17:13:26.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.663591  7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:26.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.664146  7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c [sys.catalog]: This master's current role is: FOLLOWER
17:13:26.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.700418  7080 catalog_manager.cc:1370] Generated new certificate authority record
17:13:26.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.703482  7080 catalog_manager.cc:1504] Loading token signing keys...
17:13:26.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.737272  7080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5: Generated new TSK 0
17:13:26.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.738466  7080 catalog_manager.cc:1514] Initializing in-progress tserver states...
17:13:26.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.770484  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:26.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:0
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:26.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.889712  6889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } attempt: 7
17:13:26.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.924739  6889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773: Bootstrap starting.
17:13:26.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.934064  6889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773: Neither blocks nor log segments found. Creating new log.
17:13:26.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.936975  6889 log.cc:824] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773: Log is configured to *not* fsync() on all Append() calls
17:13:26.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:26.957955  6889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773: No bootstrap required, opened a new log
17:13:27.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.007195  6889 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:27.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.008070  6889 raft_consensus.cc:374] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
17:13:27.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.008339  6889 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa10a1a0d7f4013a55cdf4770819773, State: Initialized, Role: FOLLOWER
17:13:27.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.009411  6889 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } }
17:13:27.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.013979  6889 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [sys.catalog]: configured and running, proceeding with master startup.
17:13:27.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.014178  6863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa10a1a0d7f4013a55cdf4770819773"
17:13:27.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.015013  6863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [term 0 FOLLOWER]: Advancing to term 1
17:13:27.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.019505  6864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa10a1a0d7f4013a55cdf4770819773" is_pre_election: true
17:13:27.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.039819  6863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f61cfe945c284afa8b5ddb9ed0c865b5 in term 1.
17:13:27.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.013664  7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:27.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.041133  7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [sys.catalog]: This master's current role is: FOLLOWER
17:13:27.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.041707  6862 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [term 1 FOLLOWER]: Refusing update from remote peer f61cfe945c284afa8b5ddb9ed0c865b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
17:13:27.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.049206  7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
17:13:27.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.131217  7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61cfe945c284afa8b5ddb9ed0c865b5. Latest consensus state: current_term: 1 leader_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:27.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.131349  7101 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:27.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.132102  7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [sys.catalog]: This master's current role is: FOLLOWER
17:13:27.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.132143  7101 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773 [sys.catalog]: This master's current role is: FOLLOWER
17:13:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.163398  7108 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
17:13:27.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.242080  7089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:27.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.242624  7089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:27.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.243307  7089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.276296  7089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.277313  7089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.65
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.314643  7089 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:0
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:27.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.316236  7089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:27.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.318336  7089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:27.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.342000  7115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:27.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.592489  7082 catalog_manager.cc:1259] Loaded cluster ID: 8118dd94be084fc8af6ea0af087d19cc
17:13:27.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.593108  7082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: loading cluster ID for follower catalog manager: success
17:13:27.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.600803  7082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: acquiring CA information for follower catalog manager: success
17:13:27.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.603878  7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa10a1a0d7f4013a55cdf4770819773" member_type: VOTER last_known_addr { host: "127.6.166.126" port: 39943 } } peers { permanent_uuid: "09b11d4016dc4cdcab5ca01381b23b3c" member_type: VOTER last_known_addr { host: "127.6.166.125" port: 39419 } } peers { permanent_uuid: "f61cfe945c284afa8b5ddb9ed0c865b5" member_type: VOTER last_known_addr { host: "127.6.166.124" port: 44315 } } }
17:13:27.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.605883  7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61cfe945c284afa8b5ddb9ed0c865b5 [sys.catalog]: This master's current role is: LEADER
17:13:27.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.611951  7082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09b11d4016dc4cdcab5ca01381b23b3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
17:13:28.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.190656  7108 catalog_manager.cc:1259] Loaded cluster ID: 8118dd94be084fc8af6ea0af087d19cc
17:13:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.191076  7108 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773: loading cluster ID for follower catalog manager: success
17:13:28.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.219796  7108 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773: acquiring CA information for follower catalog manager: success
17:13:28.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.228811  7108 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efa10a1a0d7f4013a55cdf4770819773: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
17:13:28.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:27.345062  7089 server_base.cc:1034] running on GCE node
17:13:28.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.345647  7114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:28.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:27.343514  7117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:28.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.663019  7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:28.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.665844  7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:28.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.667514  7089 hybrid_clock.cc:648] HybridClock initialized: now 1783530808667444 us; error 87 us; skew 500 ppm
17:13:28.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.668349  7089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:28.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.679128  7089 webserver.cc:460] Webserver started at http://127.6.166.65:35931/ using document root <none> and password file <none>
17:13:28.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.680212  7089 fs_manager.cc:359] Metadata directory not provided
17:13:28.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.680467  7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:28.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.680989  7089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
17:13:28.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.686072  7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/instance:
17:13:28.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:13:28.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:28 on dist-test-slave-v72b"
17:13:28.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.687476  7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal/instance:
17:13:28.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:13:28.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:28 on dist-test-slave-v72b"
17:13:28.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.695598  7089 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.003s	sys 0.005s
17:13:28.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.702029  7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.703279  7089 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.006s	sys 0.001s
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.703712  7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:28 on dist-test-slave-v72b"
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.704078  7089 fs_report.cc:383] FS layout report
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/data
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:28.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.765154  7089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:28.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.768182  7089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:28.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.769276  7089 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:28.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.773834  7089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:13:28.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.779309  7089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
17:13:28.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.779608  7089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
17:13:28.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.779973  7089 ts_tablet_manager.cc:610] Registered 0 tablets
17:13:28.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:28.780175  7089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
17:13:29.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.014331  7089 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.65:45345
17:13:29.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.014457  7241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.65:45345 every 8 connection(s)
17:13:29.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.019032  7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:29.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.021888  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 7089
17:13:29.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.022346  6809 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal/instance
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.036873  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:29.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:0
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:29.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:29.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.137584  7242 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:13:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.138094  7242 heartbeater.cc:464] Registering TS with master...
17:13:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.139756  7242 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:13:29.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.143097  7007 ts_manager.cc:194] Registered new tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:29.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.145941  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.65:43497
17:13:29.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.158846  7249 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:13:29.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.159268  7249 heartbeater.cc:464] Registering TS with master...
17:13:29.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.159986  7249 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:13:29.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.163178  6854 ts_manager.cc:194] Registered new tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:29.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.170504  7243 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:13:29.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.170861  7243 heartbeater.cc:464] Registering TS with master...
17:13:29.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.171588  7243 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:13:29.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.175177  6929 ts_manager.cc:194] Registered new tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:29.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:29.591092  7244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:29.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:29.591786  7244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:29.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:29.592552  7244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:29.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:29.649641  7244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:29.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:29.651239  7244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.66
17:13:29.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.720186  7244 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:29.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:0
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:29.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.721858  7244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:29.724453  7244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:29.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:29.755764  7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:30.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:30.154613  7242 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:13:31.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:29.755813  7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:31.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.144105  7257 debug-util.cc:398] Leaking SignalData structure 0x7b0800014860 after lost signal to thread 7244
17:13:31.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.288697  7244 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 1.534s	user 0.448s	sys 0.883s
17:13:31.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.291553  7244 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 1.537s	user 0.448s	sys 0.884s
17:13:31.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.291865  7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.295787  7244 server_base.cc:1034] running on GCE node
17:13:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.297185  7244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:31.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.299853  7244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:31.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.301280  7244 hybrid_clock.cc:648] HybridClock initialized: now 1783530811301159 us; error 126 us; skew 500 ppm
17:13:31.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.302016  7244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:31.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.308019  7244 webserver.cc:460] Webserver started at http://127.6.166.66:40641/ using document root <none> and password file <none>
17:13:31.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.308955  7244 fs_manager.cc:359] Metadata directory not provided
17:13:31.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.309202  7244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:31.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.309579  7244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
17:13:31.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.314675  7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/instance:
17:13:31.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:13:31.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:31 on dist-test-slave-v72b"
17:13:31.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.315944  7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal/instance:
17:13:31.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:13:31.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:31 on dist-test-slave-v72b"
17:13:31.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.322781  7244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
17:13:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.328591  7270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:31.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.329970  7244 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
17:13:31.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.330303  7244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:31.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:13:31.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:31 on dist-test-slave-v72b"
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.330651  7244 fs_report.cc:383] FS layout report
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/data
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:31.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:31.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.373889  7244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:31.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.375505  7244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:31.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.375941  7244 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.378556  7244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:13:31.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.383371  7244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
17:13:31.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.383598  7244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
17:13:31.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.383806  7244 ts_tablet_manager.cc:610] Registered 0 tablets
17:13:31.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.383977  7244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
17:13:31.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.532694  7244 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.66:35735
17:13:31.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.532794  7383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.66:35735 every 8 connection(s)
17:13:31.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.535491  7244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:31.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.543883  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 7244
17:13:31.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.544832  6809 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal/instance
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.557731  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:31.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:0
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:31.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:31.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.588186  7384 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:13:31.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.588724  7384 heartbeater.cc:464] Registering TS with master...
17:13:31.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.590077  7384 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:13:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.593513  7007 ts_manager.cc:194] Registered new tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:31.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.595737  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.66:40817
17:13:31.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.596494  7385 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:13:31.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.596889  7385 heartbeater.cc:464] Registering TS with master...
17:13:31.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.597854  7385 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:13:31.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.600847  6929 ts_manager.cc:194] Registered new tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:31.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.603940  7386 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:13:31.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.604285  7386 heartbeater.cc:464] Registering TS with master...
17:13:31.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.604945  7386 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:13:31.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.607605  6854 ts_manager.cc:194] Registered new tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.884089  7391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.884729  7391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:31.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.885188  7391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:31.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.916133  7391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:31.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.916937  7391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.67
17:13:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.952346  7391 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:13:31.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:0
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:31.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.953581  7391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:31.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.955191  7391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:31.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.976158  7400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:32.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:32.602092  7384 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:13:33.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.976573  7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:33.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:31.976392  7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:33.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:31.977685  7391 server_base.cc:1034] running on GCE node
17:13:33.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.196168  7391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:33.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.198876  7391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:33.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.200295  7391 hybrid_clock.cc:648] HybridClock initialized: now 1783530813200263 us; error 48 us; skew 500 ppm
17:13:33.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.201110  7391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:33.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.207204  7391 webserver.cc:460] Webserver started at http://127.6.166.67:44511/ using document root <none> and password file <none>
17:13:33.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.208148  7391 fs_manager.cc:359] Metadata directory not provided
17:13:33.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.208328  7391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:33.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.208766  7391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
17:13:33.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.213579  7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/instance:
17:13:33.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dd20240671db45b885a8a5ca2364ec72"
17:13:33.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:33 on dist-test-slave-v72b"
17:13:33.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.214804  7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal/instance:
17:13:33.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dd20240671db45b885a8a5ca2364ec72"
17:13:33.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:33 on dist-test-slave-v72b"
17:13:33.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.222009  7391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
17:13:33.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.228457  7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:33.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.229592  7391 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
17:13:33.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.230012  7391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:33.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dd20240671db45b885a8a5ca2364ec72"
17:13:33.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:33 on dist-test-slave-v72b"
17:13:33.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.230383  7391 fs_report.cc:383] FS layout report
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/data
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:33.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:33.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.273602  7391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:33.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.275097  7391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:33.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.275617  7391 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:33.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.277997  7391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:13:33.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.282095  7391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
17:13:33.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.282305  7391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
17:13:33.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.282536  7391 ts_tablet_manager.cc:610] Registered 0 tablets
17:13:33.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.282693  7391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
17:13:33.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.421802  7391 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.67:35283
17:13:33.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.421907  7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.67:35283 every 8 connection(s)
17:13:33.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.424718  7391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:13:33.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.428831  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 7391
17:13:33.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.429294  6809 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal/instance
17:13:33.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.467621  7526 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:13:33.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.468056  7526 heartbeater.cc:464] Registering TS with master...
17:13:33.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.469163  7526 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:13:33.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.469584  7527 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:13:33.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.469928  7527 heartbeater.cc:464] Registering TS with master...
17:13:33.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.470598  7527 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:13:33.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.472404  6929 ts_manager.cc:194] Registered new tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:33.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.473588  7525 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:13:33.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.473894  7525 heartbeater.cc:464] Registering TS with master...
17:13:33.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.474201  6854 ts_manager.cc:194] Registered new tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:33.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.474649  7525 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:13:33.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.476650  7007 ts_manager.cc:194] Registered new tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:33.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.478097  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.67:46827
17:13:33.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:33.491649  6809 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
17:13:33.502 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
17:13:33.503 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

17:13:33.513 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.6.166.126"
      port: 39943
    }
    bound_http_address {
      host: "127.6.166.126"
      port: 41475
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.6.166.125"
      port: 39419
    }
    bound_http_address {
      host: "127.6.166.125"
      port: 37501
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.6.166.124"
      port: 44315
    }
    bound_http_address {
      host: "127.6.166.124"
      port: 40473
    }
  }
}

17:13:33.520 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

17:13:33.524 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.6.166.65"
      port: 45345
    }
    bound_http_address {
      host: "127.6.166.65"
      port: 35931
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.6.166.66"
      port: 35735
    }
    bound_http_address {
      host: "127.6.166.66"
      port: 40641
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.6.166.67"
      port: 35283
    }
    bound_http_address {
      host: "127.6.166.67"
      port: 44511
    }
  }
}

17:13:33.524 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
17:13:33.713 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
17:13:33.909 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.6.166.125:39419(127.6.166.125:39419)] connecting to peer
17:13:33.970 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.6.166.124:44315(127.6.166.124:44315)] connecting to peer
17:13:33.974 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.6.166.126:39943(127.6.166.126:39943)] connecting to peer
17:13:34.010 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.6.166.124:44315(127.6.166.124:44315)] Successfully connected to peer
17:13:34.013 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.6.166.125:39419(127.6.166.125:39419)] Successfully connected to peer
17:13:34.096 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.6.166.126:39943(127.6.166.126:39943)] Successfully connected to peer
17:13:34.124 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.6.166.125
17:13:34.124 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.6.166.124
17:13:34.229 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.6.166.126
17:13:34.357 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x178d4918, L:/127.0.0.1:52576 - R:/127.6.166.125:39419] using SASL/PLAIN
17:13:34.357 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x5a41bb6f, L:/127.0.0.1:41360 - R:/127.6.166.124:44315] using SASL/PLAIN
17:13:34.378 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x2350e79a, L:/127.0.0.1:59322 - R:/127.6.166.126:39943] using SASL/PLAIN
17:13:34.410 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
17:13:34.412 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=562), Traces: [0ms] refreshing cache from master, [40ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.166.125:39419, [114ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.166.124:44315, [116ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.166.126:39943, [517ms] Sub RPC ConnectToMaster: received response from server master-127.6.166.125:39419: OK, [518ms] Sub RPC ConnectToMaster: received response from server master-127.6.166.124:44315: OK, [526ms] Sub RPC ConnectToMaster: received response from server master-127.6.166.126:39943: OK, deferred=null) after lookup
17:13:34.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.481540  7525 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:13:34.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.497165  7007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41360:
17:13:34.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837"
17:13:34.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
17:13:34.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
17:13:34.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
17:13:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
17:13:34.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
17:13:34.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
17:13:34.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
17:13:34.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
17:13:34.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
17:13:34.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
17:13:34.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
17:13:34.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:34.511035  7007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
17:13:34.594 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table b199ec737e3a4a2980c4897648cebc69
17:13:34.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.632475  7319 tablet_service.cc:1467] Processing CreateTablet for tablet e6a7a34859d84fae9805e99114ca43db (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837 [id=b199ec737e3a4a2980c4897648cebc69]), partition=RANGE (key) PARTITION UNBOUNDED
17:13:34.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.634200  7319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a7a34859d84fae9805e99114ca43db. 1 dirs total, 0 dirs full, 0 dirs failed
17:13:34.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.636195  7460 tablet_service.cc:1467] Processing CreateTablet for tablet e6a7a34859d84fae9805e99114ca43db (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837 [id=b199ec737e3a4a2980c4897648cebc69]), partition=RANGE (key) PARTITION UNBOUNDED
17:13:34.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.637998  7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a7a34859d84fae9805e99114ca43db. 1 dirs total, 0 dirs full, 0 dirs failed
17:13:34.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.641296  7177 tablet_service.cc:1467] Processing CreateTablet for tablet e6a7a34859d84fae9805e99114ca43db (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837 [id=b199ec737e3a4a2980c4897648cebc69]), partition=RANGE (key) PARTITION UNBOUNDED
17:13:34.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.642742  7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a7a34859d84fae9805e99114ca43db. 1 dirs total, 0 dirs full, 0 dirs failed
17:13:34.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.676306  7560 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap starting.
17:13:34.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.678707  7562 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap starting.
17:13:34.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.681663  7561 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap starting.
17:13:34.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.682991  7560 tablet_bootstrap.cc:654] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Neither blocks nor log segments found. Creating new log.
17:13:34.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.685216  7560 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Log is configured to *not* fsync() on all Append() calls
17:13:34.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.687060  7562 tablet_bootstrap.cc:654] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Neither blocks nor log segments found. Creating new log.
17:13:34.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.689591  7562 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Log is configured to *not* fsync() on all Append() calls
17:13:34.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.690765  7561 tablet_bootstrap.cc:654] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Neither blocks nor log segments found. Creating new log.
17:13:34.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.695215  7561 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Log is configured to *not* fsync() on all Append() calls
17:13:34.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.705900  7562 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: No bootstrap required, opened a new log
17:13:34.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.707370  7562 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.009s
17:13:34.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.706645  7560 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: No bootstrap required, opened a new log
17:13:34.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.708045  7560 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.013s
17:13:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.715180  7561 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: No bootstrap required, opened a new log
17:13:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.715780  7561 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent bootstrapping tablet: real 0.035s	user 0.013s	sys 0.014s
17:13:34.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.738674  7560 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.739631  7560 raft_consensus.cc:374] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
17:13:34.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.739923  7560 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Initialized, Role: FOLLOWER
17:13:34.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.740887  7560 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.747694  7561 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.748694  7561 raft_consensus.cc:374] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
17:13:34.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.749044  7561 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Initialized, Role: FOLLOWER
17:13:34.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.749383  7560 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent starting tablet: real 0.041s	user 0.032s	sys 0.008s
17:13:34.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.750200  7561 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.754439  7562 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.755359  7562 raft_consensus.cc:374] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
17:13:34.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.755645  7562 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd20240671db45b885a8a5ca2364ec72, State: Initialized, Role: FOLLOWER
17:13:34.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.758090  7561 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent starting tablet: real 0.042s	user 0.030s	sys 0.008s
17:13:34.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.756526  7562 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.765363  7562 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent starting tablet: real 0.057s	user 0.036s	sys 0.003s
17:13:34.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.777992  7566 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:13:34.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.778553  7566 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.781399  7566 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:34.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.795712  7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:13:34.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.795965  7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660" is_pre_election: true
17:13:34.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.796391  7480 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 0.
17:13:34.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.796829  7197 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 0.
17:13:34.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.797425  7273 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 
17:13:34.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.798193  7566 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Leader pre-election won for term 1
17:13:34.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.798512  7566 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
17:13:34.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.798732  7566 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 0 FOLLOWER]: Advancing to term 1
17:13:34.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.802776  7566 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.804279  7566 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 1 election: Requested vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:34.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.804992  7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72"
17:13:34.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.805285  7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:13:34.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.805447  7480 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 0 FOLLOWER]: Advancing to term 1
17:13:34.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.805677  7197 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 0 FOLLOWER]: Advancing to term 1
17:13:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.809931  7480 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 1.
17:13:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.809971  7197 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 1.
17:13:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.810781  7273 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 
17:13:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.811475  7566 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Leader election won for term 1
17:13:34.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.813124  7566 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 LEADER]: Becoming Leader. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Running, Role: LEADER
17:13:34.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.814384  7566 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:34.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:34.825496  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66). New cstate: current_term: 1 leader_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: UNKNOWN } } }
17:13:34.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:34.835759  7392 tablet.cc:2369] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
17:13:34.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:34.870499  7252 tablet.cc:2369] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Can't schedule compaction. Clean time has not been advanced past its initial value.
17:13:34.900 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table b199ec737e3a4a2980c4897648cebc69
17:13:34.921 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:34.927 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299993, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:34.929 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [9ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=null) after lookup
17:13:34.939 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:34.940 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] Successfully connected to peer
17:13:34.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:34.946444  7532 tablet.cc:2369] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Can't schedule compaction. Clean time has not been advanced past its initial value.
17:13:34.989 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xee5dbe49, L:/127.0.0.1:47064 - R:/127.6.166.66:35735] using TOKEN/null
17:13:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:35.003937  7480 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 1 FOLLOWER]: Refusing update from remote peer 13d4754fe07445a99ba61cb33e1d3b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
17:13:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:35.003934  7197 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 1 FOLLOWER]: Refusing update from remote peer 13d4754fe07445a99ba61cb33e1d3b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
17:13:35.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:35.005746  7571 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
17:13:35.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:35.006431  7566 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
17:13:35.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:35.043615  7574 mvcc.cc:205] Tried to move back new op lower bound from 7305342218238836736 to 7305342217487482880. Current Snapshot: MvccSnapshot[applied={T|T < 7305342218238836736}]
17:13:35.218 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:13:35.220 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.66:35735
17:13:35.221 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:13:35.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:35.222085  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 7244
17:13:35.251 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:35.258 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] handling channelInactive
17:13:35.259 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state READY due to: connection closed
17:13:35.260 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: connection closed
17:13:35.277 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:35.277 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:35.278 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [8ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: connection closed, [9ms] delaying RPC due to: Network error: connection closed, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [25ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@230595001(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@6d26c0c3 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@170b9f74 -> wakeup thread Time-limited test)) after lookup
17:13:35.279 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:35.281 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.281 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.282 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.297 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:35.297 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:35.298 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [8ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: connection closed, [9ms] delaying RPC due to: Network error: connection closed, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [25ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [28ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [30ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [45ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@230595001(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@6d26c0c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@170b9f74 -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989))) after lookup
17:13:35.299 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:35.300 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.300 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.300 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.336 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:35.337 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:35.338 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=86), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [8ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: connection closed, [9ms] delaying RPC due to: Network error: connection closed, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [25ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [28ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [30ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [45ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [47ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [49ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [49ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [80ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [85ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@230595001(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@6d26c0c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@170b9f74 -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801))) after lookup
17:13:35.339 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:35.341 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.341 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.341 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.395 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:35.395 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:35.396 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=144), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [8ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: connection closed, [9ms] delaying RPC due to: Network error: connection closed, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [25ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [28ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [30ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [45ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [47ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [49ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [49ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [80ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [85ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [88ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [90ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [90ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [143ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@230595001(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@6d26c0c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801) -> (continuation of Deferred@302718020 after retry RPC@90267831), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@170b9f74 -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801) -> (continuation of Deferred@302718020 after retry RPC@90267831))) after lookup
17:13:35.397 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:35.398 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.398 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.398 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.677 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:35.677 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:35.679 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=426), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [8ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: connection closed, [9ms] delaying RPC due to: Network error: connection closed, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [25ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [28ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [30ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [45ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [47ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [49ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [49ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [80ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [85ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [88ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [90ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [90ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [143ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [146ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [147ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [147ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [420ms] refreshing cache from master, [421ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [425ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@230595001(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@6d26c0c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801) -> (continuation of Deferred@302718020 after retry RPC@90267831) -> (continuation of Deferred@459900999 after retry RPC@1027710486), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@170b9f74 -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801) -> (continuation of Deferred@302718020 after retry RPC@90267831) -> (continuation of Deferred@459900999 after retry RPC@1027710486))) after lookup
17:13:35.680 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:35.682 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.682 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:35.683 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:36.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.668511  7583 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:13:36.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.668902  7583 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:36.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.670389  7584 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:13:36.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.671020  7583 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:36.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.670791  7584 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:36.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.680395  7584 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:36.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:36.687314  7131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:13:36.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:36.686564  7414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:13:36.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:36.701926  7131 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:13:36.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:36.701644  7414 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:13:36.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.706431  7479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:13:36.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.706990  7479 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 1.
17:13:36.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.708145  7132 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:13:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.709038  7583 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 1 FOLLOWER]: Leader pre-election won for term 2
17:13:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.709314  7583 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:13:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.709627  7583 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 1 FOLLOWER]: Advancing to term 2
17:13:36.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.712179  7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660" is_pre_election: true
17:13:36.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.714954  7583 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:36.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.715929  7197 raft_consensus.cc:2379] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd20240671db45b885a8a5ca2364ec72 in current term 2: Already voted for candidate 5f827e4e0adc48ca87a385c561e3c660 in this term.
17:13:36.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.716575  7583 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 2 election: Requested vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:36.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.717571  7479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72"
17:13:36.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.718056  7479 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 1 FOLLOWER]: Advancing to term 2
17:13:36.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.719457  7414 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660
17:13:36.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:36.720477  7131 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:13:36.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.722862  7479 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 2.
17:13:36.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.723307  7584 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
17:13:36.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.723783  7132 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:13:36.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.724505  7583 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Leader election won for term 2
17:13:36.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.726234  7583 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 LEADER]: Becoming Leader. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Running, Role: LEADER
17:13:36.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.727074  7583 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:36.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:36.739650  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 reported cstate change: term changed from 1 to 2, leader changed from 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66) to 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65). New cstate: current_term: 2 leader_uuid: "5f827e4e0adc48ca87a385c561e3c660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: HEALTHY } } }
17:13:37.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.122906  7131 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 -> Peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Couldn't send request to peer 13d4754fe07445a99ba61cb33e1d3b2a. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:13:37.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.192808  6885 debug-util.cc:398] Leaking SignalData structure 0x7b08000a35e0 after lost signal to thread 6822
17:13:37.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.193501  6885 debug-util.cc:398] Leaking SignalData structure 0x7b080008fac0 after lost signal to thread 6888
17:13:37.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:37.239212  7479 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Refusing update from remote peer 5f827e4e0adc48ca87a385c561e3c660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
17:13:37.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:37.241168  7583 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
17:13:37.379 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:37.379 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:37.380 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2128), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [8ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: connection closed, [9ms] delaying RPC due to: Network error: connection closed, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [25ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [28ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [30ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [45ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [47ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [49ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [49ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [80ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [85ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [88ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [90ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [90ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [143ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [146ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [147ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [147ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [420ms] refreshing cache from master, [421ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [425ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [428ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [432ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [432ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2120ms] refreshing cache from master, [2121ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [2127ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@230595001(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@6d26c0c3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801) -> (continuation of Deferred@302718020 after retry RPC@90267831) -> (continuation of Deferred@459900999 after retry RPC@1027710486) -> (continuation of Deferred@507913132 after retry RPC@1287721854), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@170b9f74 -> wakeup thread Time-limited test -> (continuation of Deferred@2084571926 after retry RPC@930859989) -> (continuation of Deferred@242227332 after retry RPC@396368801) -> (continuation of Deferred@302718020 after retry RPC@90267831) -> (continuation of Deferred@459900999 after retry RPC@1027710486) -> (continuation of Deferred@507913132 after retry RPC@1287721854))) after lookup
17:13:37.381 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:37.384 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] Successfully connected to peer
17:13:37.417 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xae95bfd2, L:/127.0.0.1:42736 - R:/127.6.166.65:45345] using TOKEN/null
17:13:37.492 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.66:35735
17:13:37.495 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:13:37.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:37.497506  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:37.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:37.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:37.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.816164  7618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:37.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.816617  7618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:37.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.817049  7618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:37.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.848506  7618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:37.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.850095  7618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.66
17:13:37.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:37.904321  7618 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:37.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:37.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:37.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:37.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:13:37.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:37.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:37.905870  7618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:37.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:37.907796  7618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.929716  7625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:39.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.932031  7627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:39.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:37.929729  7624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:39.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.550702  7618 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
17:13:39.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:39.551378  7626 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1619 milliseconds
17:13:39.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.552997  7618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:39.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.556020  7618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:39.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.557523  7618 hybrid_clock.cc:648] HybridClock initialized: now 1783530819557415 us; error 120 us; skew 500 ppm
17:13:39.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.558809  7618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:39.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:39.559719  7131 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 -> Peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Couldn't send request to peer 13d4754fe07445a99ba61cb33e1d3b2a. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
17:13:39.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.568893  7618 webserver.cc:460] Webserver started at http://127.6.166.66:40641/ using document root <none> and password file <none>
17:13:39.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.569803  7618 fs_manager.cc:359] Metadata directory not provided
17:13:39.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.570012  7618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:39.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.581506  7618 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.003s	sys 0.004s
17:13:39.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.587966  7637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:39.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.589335  7618 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.002s
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.589635  7618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:31 on dist-test-slave-v72b"
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.592201  7618 fs_report.cc:383] FS layout report
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:39.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/data
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:39.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:39.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.642493  7618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:39.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.644057  7618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:39.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.644470  7618 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:39.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.647581  7618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:13:39.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.654495  7644 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:13:39.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.662833  7618 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:13:39.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.663085  7618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.001s
17:13:39.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.663373  7618 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:13:39.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.668550  7618 ts_tablet_manager.cc:610] Registered 1 tablets
17:13:39.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.668812  7618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
17:13:39.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.669152  7644 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap starting.
17:13:39.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.735353  7644 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Log is configured to *not* fsync() on all Append() calls
17:13:39.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.771059  7644 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:13:39.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.772375  7644 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap complete.
17:13:39.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.774418  7644 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent bootstrapping tablet: real 0.106s	user 0.070s	sys 0.030s
17:13:39.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.799177  7644 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:39.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.802968  7644 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Initialized, Role: FOLLOWER
17:13:39.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.804075  7644 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:39.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.807864  7644 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent starting tablet: real 0.033s	user 0.020s	sys 0.012s
17:13:39.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.864156  7618 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.66:35735
17:13:39.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.864418  7752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.66:35735 every 8 connection(s)
17:13:39.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.867067  7618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:39.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.876920  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 7618
17:13:39.878 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:39.925 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:13:39.927 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.65:45345
17:13:39.927 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

17:13:39.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.928275  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 7089
17:13:39.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.934460  7753 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:13:39.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.934983  7753 heartbeater.cc:464] Registering TS with master...
17:13:39.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.936215  7753 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:13:39.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.937651  7755 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:13:39.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.938001  7755 heartbeater.cc:464] Registering TS with master...
17:13:39.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.938860  7755 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:13:39.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.939888  7754 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:13:39.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.940200  7754 heartbeater.cc:464] Registering TS with master...
17:13:39.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.940740  7007 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:39.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.940963  7754 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:13:39.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.943797  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.66:44591
17:13:39.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.944197  6854 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:39.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:39.945290  6929 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:39.964 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:39.964 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] handling channelInactive
17:13:39.964 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state READY due to: connection closed
17:13:39.972 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:39.975 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:39.975 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:39.975 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:39.995 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:39.996 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:39.997 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=3, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [1ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [11ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@667362865(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@3597611a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@167233bc -> wakeup thread Time-limited test)) after lookup
17:13:39.997 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:39.998 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:39.999 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:39.999 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.016 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:40.017 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:40.018 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=5, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [1ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [11ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [33ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [35ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@667362865(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@3597611a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@167233bc -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467))) after lookup
17:13:40.018 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:40.020 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.020 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.021 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.096 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:40.096 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:40.097 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=7, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=132), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [1ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [11ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [33ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [35ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [54ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [57ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [57ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@667362865(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@3597611a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@167233bc -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852))) after lookup
17:13:40.098 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:40.099 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.099 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.099 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.256 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:40.256 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:40.257 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=9, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=292), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [1ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [11ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [33ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [35ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [54ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [57ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [57ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [133ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [135ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [135ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [287ms] refreshing cache from master, [288ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [291ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@667362865(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@3597611a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852) -> (continuation of Deferred@1690068312 after retry RPC@2096352241), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@167233bc -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852) -> (continuation of Deferred@1690068312 after retry RPC@2096352241))) after lookup
17:13:40.257 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:40.259 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.259 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.259 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:40.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:40.947865  7753 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:13:41.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.135289  7766 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:13:41.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.135707  7766 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:41.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.137955  7766 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:41.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:41.142120  7639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:41.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:41.145814  7639 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:41.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.150831  7479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:13:41.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.152268  7640 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a; no voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72
17:13:41.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.152964  7766 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
17:13:41.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.195614  7771 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5f827e4e0adc48ca87a385c561e3c660)
17:13:41.196 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:41.196 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:41.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.196084  7771 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:41.197 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=11, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1232), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [1ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [11ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [33ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [35ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [54ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [57ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [57ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [133ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [135ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [135ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [287ms] refreshing cache from master, [288ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [291ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [293ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [295ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [295ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [1227ms] refreshing cache from master, [1227ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1231ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@667362865(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@3597611a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852) -> (continuation of Deferred@1690068312 after retry RPC@2096352241) -> (continuation of Deferred@1175028526 after retry RPC@617047133), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@167233bc -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852) -> (continuation of Deferred@1690068312 after retry RPC@2096352241) -> (continuation of Deferred@1175028526 after retry RPC@617047133))) after lookup
17:13:41.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.197952  7771 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:41.198 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:41.203 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:41.203 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:41.203 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:41.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:41.212224  7414 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:41.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.217829  7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" is_pre_election: true
17:13:41.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.218568  7707 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd20240671db45b885a8a5ca2364ec72 in term 1.
17:13:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.219913  7414 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 5f827e4e0adc48ca87a385c561e3c660
17:13:41.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.220515  7771 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Leader pre-election won for term 3
17:13:41.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.220818  7771 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5f827e4e0adc48ca87a385c561e3c660)
17:13:41.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.221127  7771 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 2 FOLLOWER]: Advancing to term 3
17:13:41.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.225282  7771 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:41.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.226647  7771 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 3 election: Requested vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:41.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.227628  7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:13:41.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.228200  7707 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 1 FOLLOWER]: Advancing to term 3
17:13:41.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:41.230129  7414 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.235533  7707 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd20240671db45b885a8a5ca2364ec72 in term 3.
17:13:41.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.236397  7414 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 5f827e4e0adc48ca87a385c561e3c660
17:13:41.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.237135  7771 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 3 FOLLOWER]: Leader election won for term 3
17:13:41.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.238790  7771 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 3 LEADER]: Becoming Leader. State: Replica: dd20240671db45b885a8a5ca2364ec72, State: Running, Role: LEADER
17:13:41.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.240062  7771 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:41.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.248379  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 reported cstate change: term changed from 2 to 3, leader changed from 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65) to dd20240671db45b885a8a5ca2364ec72 (127.6.166.67). New cstate: current_term: 3 leader_uuid: "dd20240671db45b885a8a5ca2364ec72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: UNKNOWN } } }
17:13:41.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.777576  7707 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 3 FOLLOWER]: Refusing update from remote peer dd20240671db45b885a8a5ca2364ec72: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
17:13:41.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:41.778966  7788 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
17:13:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:41.809765  7414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111) [suppressed 2 similar messages]
17:13:41.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:41.816733  7414 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 -> Peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Couldn't send request to peer 5f827e4e0adc48ca87a385c561e3c660. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:13:44.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:44.338526  7414 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 -> Peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Couldn't send request to peer 5f827e4e0adc48ca87a385c561e3c660. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
17:13:44.596 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:44.597 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:13:44.598 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=13, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=4633), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [1ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [11ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [33ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [35ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [54ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [57ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [57ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [133ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [135ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [135ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [287ms] refreshing cache from master, [288ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [291ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [293ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [295ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [295ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [1227ms] refreshing cache from master, [1227ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1231ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [1234ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [1239ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [1239ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [4627ms] refreshing cache from master, [4627ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [4631ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@667362865(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@3597611a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852) -> (continuation of Deferred@1690068312 after retry RPC@2096352241) -> (continuation of Deferred@1175028526 after retry RPC@617047133) -> (continuation of Deferred@1964396539 after retry RPC@793515049), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@167233bc -> wakeup thread Time-limited test -> (continuation of Deferred@358905250 after retry RPC@1881850467) -> (continuation of Deferred@11767147 after retry RPC@648838852) -> (continuation of Deferred@1690068312 after retry RPC@2096352241) -> (continuation of Deferred@1175028526 after retry RPC@617047133) -> (continuation of Deferred@1964396539 after retry RPC@793515049))) after lookup
17:13:44.599 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:44.601 [DEBUG - kudu-nio-0] (Connection.java:757) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] Successfully connected to peer
17:13:44.630 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xa540f30e, L:/127.0.0.1:49798 - R:/127.6.166.67:35283] using TOKEN/null
17:13:44.690 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.65:45345
17:13:44.691 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:44.693192  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:45345
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35931
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:44.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:45.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.026005  7818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:45.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.026578  7818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:45.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.027179  7818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:45.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.063228  7818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:45.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.064245  7818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.65
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:45.099020  7818 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:45345
17:13:45.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35931
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:45.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:45.100579  7818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:45.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:45.102239  7818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:45.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.121850  7827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:46.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.121896  7825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:46.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:45.123669  7818 server_base.cc:1034] running on GCE node
17:13:46.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:45.121850  7824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:46.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.369155  7818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:46.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.371331  7818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:46.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.372690  7818 hybrid_clock.cc:648] HybridClock initialized: now 1783530826372637 us; error 69 us; skew 500 ppm
17:13:46.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.373555  7818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:46.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.383496  7818 webserver.cc:460] Webserver started at http://127.6.166.65:35931/ using document root <none> and password file <none>
17:13:46.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.384531  7818 fs_manager.cc:359] Metadata directory not provided
17:13:46.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.384743  7818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:46.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.392560  7818 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.001s
17:13:46.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.397334  7838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:46.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.398340  7818 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.001s
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.398705  7818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:28 on dist-test-slave-v72b"
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.401160  7818 fs_report.cc:383] FS layout report
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/data
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:46.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:46.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:46.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.440542  7818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:46.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.442057  7818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:46.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.442561  7818 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:46.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.445151  7818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:13:46.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.451438  7845 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:13:46.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.459524  7818 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:13:46.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.459803  7818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
17:13:46.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.460153  7818 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:13:46.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.464915  7818 ts_tablet_manager.cc:610] Registered 1 tablets
17:13:46.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.465173  7818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
17:13:46.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.465535  7845 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap starting.
17:13:46.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.538071  7845 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Log is configured to *not* fsync() on all Append() calls
17:13:46.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.574812  7845 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:13:46.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.575685  7845 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap complete.
17:13:46.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.577083  7845 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent bootstrapping tablet: real 0.112s	user 0.081s	sys 0.027s
17:13:46.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.597584  7845 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:46.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.600033  7845 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Initialized, Role: FOLLOWER
17:13:46.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.601485  7845 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:46.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.606196  7845 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent starting tablet: real 0.029s	user 0.013s	sys 0.016s
17:13:46.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.643380  7818 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.65:45345
17:13:46.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.643589  7953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.65:45345 every 8 connection(s)
17:13:46.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.645751  7818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:46.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.648486  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 7818
17:13:46.649 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:46.682 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:13:46.684 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.67:35283
17:13:46.684 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

17:13:46.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.684872  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 7391
17:13:46.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.699472  7955 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:13:46.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.699952  7955 heartbeater.cc:464] Registering TS with master...
17:13:46.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.701095  7955 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:13:46.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.703702  7954 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:13:46.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.704061  7954 heartbeater.cc:464] Registering TS with master...
17:13:46.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.704908  7954 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:13:46.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.705715  6929 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:46.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.710007  7006 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:46.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.713487  7006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.65:45157
17:13:46.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.720091  7956 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:13:46.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.720469  7956 heartbeater.cc:464] Registering TS with master...
17:13:46.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.721170  7956 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:13:46.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:46.724388  6854 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:46.726 [DEBUG - kudu-nio-0] (Connection.java:239) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] handling channelInactive
17:13:46.726 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state READY due to: connection closed
17:13:46.728 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:46.728 [DEBUG - Time-limited test] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:46.736 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.736 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.738 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.756 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:46.756 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:13:46.756 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [8ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [27ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1923825399(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@22b7dc20 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42dba164 -> wakeup thread Time-limited test)) after lookup
17:13:46.757 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:46.759 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.759 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.759 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.775 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:46.775 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:13:46.776 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [8ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [27ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [29ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [31ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [46ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1923825399(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@22b7dc20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42dba164 -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460))) after lookup
17:13:46.776 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:46.777 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.777 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.777 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.795 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:46.796 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:13:46.796 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [8ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [27ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [29ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [31ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [46ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [48ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [49ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [49ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [67ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1923825399(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@22b7dc20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42dba164 -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153))) after lookup
17:13:46.797 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:46.798 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.798 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.798 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.855 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:46.856 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:13:46.856 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=128), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [8ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [27ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [29ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [31ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [46ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [48ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [49ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [49ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [67ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [68ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [70ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [70ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [127ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1923825399(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@22b7dc20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153) -> (continuation of Deferred@1349387681 after retry RPC@1477349213), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42dba164 -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153) -> (continuation of Deferred@1349387681 after retry RPC@1477349213))) after lookup
17:13:46.857 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:46.859 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.859 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:46.859 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:47.256 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:47.256 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:13:47.257 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=528), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [8ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [27ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [29ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [31ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [46ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [48ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [49ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [49ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [67ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [68ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [70ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [70ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [127ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [129ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [131ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [131ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [523ms] refreshing cache from master, [524ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [527ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1923825399(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@22b7dc20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153) -> (continuation of Deferred@1349387681 after retry RPC@1477349213) -> (continuation of Deferred@188372792 after retry RPC@883409082), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42dba164 -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153) -> (continuation of Deferred@1349387681 after retry RPC@1477349213) -> (continuation of Deferred@188372792 after retry RPC@883409082))) after lookup
17:13:47.258 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:47.260 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:47.260 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:47.260 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:13:47.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:47.342859  6960 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1b40 after lost signal to thread 6895
17:13:47.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:47.717851  7954 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:13:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.042490  7966 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:13:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.042965  7966 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:48.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.045424  7966 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:48.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:48.052520  7841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:13:48.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:48.056851  7841 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:13:48.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.063233  7707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" is_pre_election: true
17:13:48.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.065541  7840 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660; no voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72
17:13:48.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.066478  7966 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
17:13:48.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.142771  7971 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd20240671db45b885a8a5ca2364ec72)
17:13:48.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.143299  7971 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:48.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.145253  7971 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:48.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:48.160889  7640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:13:48.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:48.166344  7640 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:13:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.173316  7908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660" is_pre_election: true
17:13:48.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.174046  7908 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 2.
17:13:48.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.175575  7639 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660; no voters: dd20240671db45b885a8a5ca2364ec72
17:13:48.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.176342  7971 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
17:13:48.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.176645  7971 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd20240671db45b885a8a5ca2364ec72)
17:13:48.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.176865  7971 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 3 FOLLOWER]: Advancing to term 4
17:13:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.181460  7971 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:48.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.182900  7971 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 4 election: Requested vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:48.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.184017  7908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:13:48.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.184566  7908 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 2 FOLLOWER]: Advancing to term 4
17:13:48.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:48.186964  7640 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:13:48.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.191677  7908 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 4.
17:13:48.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.192621  7639 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660; no voters: dd20240671db45b885a8a5ca2364ec72
17:13:48.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.193287  7971 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Leader election won for term 4
17:13:48.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.194645  7971 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 LEADER]: Becoming Leader. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Running, Role: LEADER
17:13:48.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.195708  7971 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:48.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.206053  7006 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a reported cstate change: term changed from 3 to 4, leader changed from dd20240671db45b885a8a5ca2364ec72 (127.6.166.67) to 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66). New cstate: current_term: 4 leader_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: UNKNOWN } } }
17:13:48.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.597584  7908 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 4 FOLLOWER]: Refusing update from remote peer 13d4754fe07445a99ba61cb33e1d3b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
17:13:48.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.599066  7971 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
17:13:48.616 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:48.616 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:48.617 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1888), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [8ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [27ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [29ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [31ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [46ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [48ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [49ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [49ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [67ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [68ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [70ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [70ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [123ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [127ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [129ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [131ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [131ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [523ms] refreshing cache from master, [524ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [527ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [530ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [532ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [532ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [1883ms] refreshing cache from master, [1883ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1887ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1923825399(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@22b7dc20 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153) -> (continuation of Deferred@1349387681 after retry RPC@1477349213) -> (continuation of Deferred@188372792 after retry RPC@883409082) -> (continuation of Deferred@21917736 after retry RPC@191283883), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42dba164 -> wakeup thread Time-limited test -> (continuation of Deferred@1336087701 after retry RPC@1357667460) -> (continuation of Deferred@838154917 after retry RPC@1844224153) -> (continuation of Deferred@1349387681 after retry RPC@1477349213) -> (continuation of Deferred@188372792 after retry RPC@883409082) -> (continuation of Deferred@21917736 after retry RPC@191283883))) after lookup
17:13:48.618 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:48.619 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] Successfully connected to peer
17:13:48.682 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x059eb9d8, L:/127.0.0.1:37550 - R:/127.6.166.66:35735] using TOKEN/null
17:13:48.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:48.702574  7640 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a -> Peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Couldn't send request to peer dd20240671db45b885a8a5ca2364ec72. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:13:48.832 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.67:35283
17:13:48.832 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:48.833678  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:13:48.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:35283
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44511
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:49.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.205652  7999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:49.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.206110  7999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:49.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.206525  7999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:49.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.238121  7999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:49.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.238996  7999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.67
17:13:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:49.276304  7999 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:13:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:35283
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44511
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:49.277417  7999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:49.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:49.278842  7999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:49.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.300458  8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:50.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.300796  8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:50.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:49.300484  8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:50.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:50.541623  8007 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
17:13:50.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.542166  7999 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
17:13:50.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.545861  7999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:50.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.548632  7999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:50.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.550069  7999 hybrid_clock.cc:648] HybridClock initialized: now 1783530830550024 us; error 64 us; skew 500 ppm
17:13:50.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.550858  7999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:50.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.557535  7999 webserver.cc:460] Webserver started at http://127.6.166.67:44511/ using document root <none> and password file <none>
17:13:50.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.558611  7999 fs_manager.cc:359] Metadata directory not provided
17:13:50.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.558885  7999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:50.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.566928  7999 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
17:13:50.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.571715  8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:50.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.573009  7999 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
17:13:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.573309  7999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dd20240671db45b885a8a5ca2364ec72"
17:13:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:33 on dist-test-slave-v72b"
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.575677  7999 fs_report.cc:383] FS layout report
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/data
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:50.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.612825  7999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:50.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.614315  7999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:50.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.614723  7999 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:50.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.617308  7999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:13:50.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.623147  8027 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:13:50.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.630714  7999 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:13:50.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.630945  7999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s	user 0.002s	sys 0.000s
17:13:50.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.631215  7999 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:13:50.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.636183  7999 ts_tablet_manager.cc:610] Registered 1 tablets
17:13:50.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.636436  7999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
17:13:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.636757  8027 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap starting.
17:13:50.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.699769  8027 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Log is configured to *not* fsync() on all Append() calls
17:13:50.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.763218  8027 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:13:50.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.764292  8027 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap complete.
17:13:50.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.766408  8027 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent bootstrapping tablet: real 0.130s	user 0.103s	sys 0.024s
17:13:50.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.797420  8027 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:50.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.801710  8027 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd20240671db45b885a8a5ca2364ec72, State: Initialized, Role: FOLLOWER
17:13:50.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.803004  8027 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:50.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.808342  8027 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent starting tablet: real 0.042s	user 0.037s	sys 0.004s
17:13:50.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.849077  7999 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.67:35283
17:13:50.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.849244  8135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.67:35283 every 8 connection(s)
17:13:50.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.851639  7999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:13:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.855438  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 7999
17:13:50.856 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:50.893 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:13:50.908 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.66:35735
17:13:50.908 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:13:50.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.909152  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 7618
17:13:50.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.914001  8136 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:13:50.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.914474  8136 heartbeater.cc:464] Registering TS with master...
17:13:50.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.915617  8136 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:13:50.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.919866  7006 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:50.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.923919  7006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.67:40003
17:13:50.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.937309  8138 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:13:50.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.937708  8138 heartbeater.cc:464] Registering TS with master...
17:13:50.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.938510  8138 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:13:50.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.941862  8137 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:13:50.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.942245  8137 heartbeater.cc:464] Registering TS with master...
17:13:50.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.942972  8137 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:13:50.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.943490  6854 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:50.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:50.947151  6929 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:50.957 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] handling channelInactive
17:13:50.958 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state READY due to: connection closed
17:13:50.959 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:50.960 [DEBUG - Time-limited test] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:50.962 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.962 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.963 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.976 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:50.976 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:50.977 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [3ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [3ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [15ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1561726996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2e7972ce -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e7f50ca -> wakeup thread Time-limited test)) after lookup
17:13:50.977 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:50.978 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.978 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.978 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.994 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:50.995 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:50.995 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [3ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [3ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [15ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [17ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [18ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [18ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [34ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1561726996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2e7972ce -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e7f50ca -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810))) after lookup
17:13:50.996 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:50.997 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.997 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:50.998 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.036 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:51.036 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:51.037 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [3ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [3ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [15ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [17ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [18ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [18ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [34ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [35ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [38ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [38ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [75ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1561726996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2e7972ce -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e7f50ca -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709))) after lookup
17:13:51.038 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:51.039 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.039 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.039 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.156 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:51.156 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:51.157 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=197), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [3ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [3ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [15ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [17ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [18ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [18ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [34ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [35ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [38ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [38ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [75ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [77ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [79ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [79ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [195ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1561726996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2e7972ce -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709) -> (continuation of Deferred@835727047 after retry RPC@153525395), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e7f50ca -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709) -> (continuation of Deferred@835727047 after retry RPC@153525395))) after lookup
17:13:51.157 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:51.159 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.160 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.160 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:51.335 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:51.336 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=376), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [3ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [3ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [15ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [17ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [18ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [18ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [34ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [35ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [38ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [38ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [75ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [77ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [79ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [79ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [195ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [197ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [200ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [200ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [371ms] refreshing cache from master, [371ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [375ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1561726996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2e7972ce -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709) -> (continuation of Deferred@835727047 after retry RPC@153525395) -> (continuation of Deferred@1560750894 after retry RPC@1626928665), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e7f50ca -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709) -> (continuation of Deferred@835727047 after retry RPC@153525395) -> (continuation of Deferred@1560750894 after retry RPC@1626928665))) after lookup
17:13:51.337 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:13:51.338 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.338 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.338 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:13:51.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.886411  8148 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:13:51.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.886790  8148 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:51.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.888310  8148 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:51.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:51.893499  7840 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:13:51.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.901321  8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:13:51.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.902210  8090 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 3.
17:13:51.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.903537  7841 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:13:51.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.904173  8148 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 4 FOLLOWER]: Leader pre-election won for term 5
17:13:51.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.904480  8148 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:13:51.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.904805  8148 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 4 FOLLOWER]: Advancing to term 5
17:13:51.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.909495  8148 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.910998  8148 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 5 election: Requested vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:13:51.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.912107  8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72"
17:13:51.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.912671  8090 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 3 FOLLOWER]: Advancing to term 5
17:13:51.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:51.914366  7840 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:13:51.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.919941  8090 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 5.
17:13:51.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.920915  7841 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:13:51.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.921743  8148 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Leader election won for term 5
17:13:51.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.923177  8148 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 LEADER]: Becoming Leader. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Running, Role: LEADER
17:13:51.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.924203  8148 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:51.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.928731  8136 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:13:51.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:51.934283  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 reported cstate change: term changed from 4 to 5, leader changed from 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66) to 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65). New cstate: current_term: 5 leader_uuid: "5f827e4e0adc48ca87a385c561e3c660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: HEALTHY } } }
17:13:52.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:52.410552  7840 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 -> Peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Couldn't send request to peer 13d4754fe07445a99ba61cb33e1d3b2a. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:13:52.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:52.453202  8090 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 5 FOLLOWER]: Refusing update from remote peer 5f827e4e0adc48ca87a385c561e3c660: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
17:13:52.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:52.454672  8148 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
17:13:52.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:52.470012  8166 mvcc.cc:205] Tried to move back new op lower bound from 7305342274424029184 to 7305342272298110976. Current Snapshot: MvccSnapshot[applied={T|T < 7305342274424029184}]
17:13:52.995 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:52.995 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:52.996 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2036), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [3ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [3ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [15ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [17ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [18ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [18ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [34ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [35ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [38ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [38ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [75ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [77ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [79ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [79ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [195ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [197ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [200ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [200ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [371ms] refreshing cache from master, [371ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [375ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [376ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [378ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [378ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2031ms] refreshing cache from master, [2031ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [2034ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1561726996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2e7972ce -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709) -> (continuation of Deferred@835727047 after retry RPC@153525395) -> (continuation of Deferred@1560750894 after retry RPC@1626928665) -> (continuation of Deferred@314093704 after retry RPC@779252262), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e7f50ca -> wakeup thread Time-limited test -> (continuation of Deferred@1995884875 after retry RPC@1905412810) -> (continuation of Deferred@692109045 after retry RPC@1336126709) -> (continuation of Deferred@835727047 after retry RPC@153525395) -> (continuation of Deferred@1560750894 after retry RPC@1626928665) -> (continuation of Deferred@314093704 after retry RPC@779252262))) after lookup
17:13:52.997 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:52.998 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] Successfully connected to peer
17:13:53.039 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xcd5e9010, L:/127.0.0.1:54556 - R:/127.6.166.65:45345] using TOKEN/null
17:13:53.083 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.66:35735
17:13:53.084 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:13:53.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:53.085377  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:53.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:53.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:53.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:13:53.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:53.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:13:53.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:53.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:13:53.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:13:53.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:53.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:53.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:53.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:53.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.346423  7840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111) [suppressed 4 similar messages]
17:13:53.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.399899  8181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:53.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.400471  8181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:53.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.401064  8181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.434214  8181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:53.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.435164  8181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.66
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:53.470945  8181 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:13:53.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:53.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:53.472512  8181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:53.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:53.474195  8181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:53.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.501173  8189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:54.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.501255  8190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:54.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:53.503957  8181 server_base.cc:1034] running on GCE node
17:13:54.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:53.502022  8192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:54.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.712268  8181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:13:54.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.715086  8181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:13:54.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.716576  8181 hybrid_clock.cc:648] HybridClock initialized: now 1783530834716552 us; error 50 us; skew 500 ppm
17:13:54.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.717497  8181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:13:54.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.734898  8181 webserver.cc:460] Webserver started at http://127.6.166.66:40641/ using document root <none> and password file <none>
17:13:54.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.736256  8181 fs_manager.cc:359] Metadata directory not provided
17:13:54.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.736548  8181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:13:54.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.746099  8181 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
17:13:54.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.750771  8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:13:54.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.751859  8181 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
17:13:54.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.752223  8181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:54.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:13:54.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:31 on dist-test-slave-v72b"
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.754722  8181 fs_report.cc:383] FS layout report
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/data
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:13:54.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:13:54.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.789844  8181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:13:54.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.791531  8181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:13:54.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.792038  8181 kserver.cc:163] Server-wide thread pool size limit: 3276
17:13:54.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.795284  8181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:13:54.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.801066  8209 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:13:54.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.808655  8181 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:13:54.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.808993  8181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s	user 0.002s	sys 0.000s
17:13:54.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.809324  8181 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:13:54.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.814170  8181 ts_tablet_manager.cc:610] Registered 1 tablets
17:13:54.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.814338  8181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
17:13:54.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.814836  8209 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap starting.
17:13:54.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:54.832167  7840 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 -> Peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Couldn't send request to peer 13d4754fe07445a99ba61cb33e1d3b2a. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
17:13:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.881253  8209 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Log is configured to *not* fsync() on all Append() calls
17:13:54.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.950630  8209 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:13:54.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.951635  8209 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap complete.
17:13:54.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.953274  8209 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent bootstrapping tablet: real 0.139s	user 0.099s	sys 0.031s
17:13:54.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.972978  8209 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:54.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.975703  8209 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Initialized, Role: FOLLOWER
17:13:54.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.976905  8209 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:54.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:54.980543  8209 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent starting tablet: real 0.027s	user 0.019s	sys 0.009s
17:13:55.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.033476  8181 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.66:35735
17:13:55.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.033627  8318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.66:35735 every 8 connection(s)
17:13:55.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.036028  8181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:13:55.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.044319  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 8181
17:13:55.045 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:55.053 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
17:13:55.055 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
17:13:55.088 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:13:55.090 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.65:45345
17:13:55.090 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

17:13:55.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.090916  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 7818
17:13:55.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.093778  8321 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:13:55.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.094280  8321 heartbeater.cc:464] Registering TS with master...
17:13:55.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.095466  8321 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:13:55.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.099927  6854 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:55.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.100893  8319 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:13:55.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.101204  8319 heartbeater.cc:464] Registering TS with master...
17:13:55.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.102032  8319 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:13:55.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.104173  8320 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:13:55.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.104535  8320 heartbeater.cc:464] Registering TS with master...
17:13:55.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.105059  7007 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:55.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.105440  8320 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:13:55.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.107775  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.66:59597
17:13:55.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:55.109187  6929 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:13:55.123 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] handling channelInactive
17:13:55.123 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state READY due to: connection closed
17:13:55.123 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:13:55.124 [DEBUG - Time-limited test] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:55.126 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.126 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.126 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.134 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:55.134 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:55.135 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [2ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@573717387(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@56ef25a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b41cfc3 -> wakeup thread Time-limited test)) after lookup
17:13:55.136 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:55.137 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.137 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.137 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.155 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:55.155 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:55.155 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [2ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [11ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [13ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [14ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@573717387(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@56ef25a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b41cfc3 -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237))) after lookup
17:13:55.156 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:55.157 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.157 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.157 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.175 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:55.176 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:55.176 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [2ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [11ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [13ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [14ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [32ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [33ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [33ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@573717387(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@56ef25a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b41cfc3 -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566))) after lookup
17:13:55.177 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:55.178 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.178 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.178 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.256 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:55.256 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:55.257 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=132), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [2ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [11ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [13ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [14ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [32ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [33ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [33ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [52ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [54ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [55ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@573717387(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@56ef25a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b41cfc3 -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006))) after lookup
17:13:55.258 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:55.259 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.259 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.259 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.655 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:55.656 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:55.656 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=532), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [2ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [11ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [13ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [14ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [32ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [33ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [33ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [52ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [54ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [55ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [133ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [135ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [135ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [528ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [531ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@573717387(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@56ef25a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006) -> (continuation of Deferred@721822915 after retry RPC@381678170), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b41cfc3 -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006) -> (continuation of Deferred@721822915 after retry RPC@381678170))) after lookup
17:13:55.657 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:55.660 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.660 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:55.661 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:56.035 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:56.036 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:13:56.037 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=912), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [2ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [11ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [13ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [14ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [32ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [33ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [33ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [52ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [54ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [55ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [133ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [135ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [135ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [528ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [531ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [532ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [536ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [537ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [907ms] refreshing cache from master, [907ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [910ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@573717387(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@56ef25a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006) -> (continuation of Deferred@721822915 after retry RPC@381678170) -> (continuation of Deferred@338374110 after retry RPC@2112823331), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b41cfc3 -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006) -> (continuation of Deferred@721822915 after retry RPC@381678170) -> (continuation of Deferred@338374110 after retry RPC@2112823331))) after lookup
17:13:56.038 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:13:56.039 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:56.039 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:56.039 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:13:56.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.112375  8319 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:13:56.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.331118  8332 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:13:56.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.331672  8332 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:56.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.334009  8332 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:56.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:56.337472  8204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:56.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:56.341800  8204 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:56.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.345896  8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:13:56.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.347651  8205 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a; no voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72
17:13:56.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.348685  8332 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
17:13:56.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.532186  8337 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f827e4e0adc48ca87a385c561e3c660)
17:13:56.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.532552  8337 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:56.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.534651  8337 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:56.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:56.538374  8022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:56.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:56.543021  8022 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:56.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.546708  8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" is_pre_election: true
17:13:56.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.547384  8272 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd20240671db45b885a8a5ca2364ec72 in term 4.
17:13:56.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.548514  8022 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 5f827e4e0adc48ca87a385c561e3c660
17:13:56.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.549222  8337 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 5 FOLLOWER]: Leader pre-election won for term 6
17:13:56.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.549446  8337 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5f827e4e0adc48ca87a385c561e3c660)
17:13:56.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.549676  8337 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 5 FOLLOWER]: Advancing to term 6
17:13:56.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.553735  8337 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:56.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.555059  8337 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 6 election: Requested vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:13:56.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.555955  8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:13:56.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.556514  8272 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 4 FOLLOWER]: Advancing to term 6
17:13:56.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:56.558604  8022 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:13:56.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.563401  8272 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd20240671db45b885a8a5ca2364ec72 in term 6.
17:13:56.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.564211  8022 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 5f827e4e0adc48ca87a385c561e3c660
17:13:56.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.564875  8337 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Leader election won for term 6
17:13:56.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.566411  8337 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 LEADER]: Becoming Leader. State: Replica: dd20240671db45b885a8a5ca2364ec72, State: Running, Role: LEADER
17:13:56.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.567305  8337 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:13:56.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:56.575742  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 reported cstate change: term changed from 5 to 6, leader changed from 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65) to dd20240671db45b885a8a5ca2364ec72 (127.6.166.67). New cstate: current_term: 6 leader_uuid: "dd20240671db45b885a8a5ca2364ec72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: UNKNOWN } } }
17:13:56.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:56.986320  8022 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 -> Peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Couldn't send request to peer 5f827e4e0adc48ca87a385c561e3c660. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:13:57.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:57.037047  8272 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 6 FOLLOWER]: Refusing update from remote peer dd20240671db45b885a8a5ca2364ec72: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
17:13:57.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:57.038887  8337 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
17:13:58.958 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:13:58.958 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:13:58.959 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=14, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=3835), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [2ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [6ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [10ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [11ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [13ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [14ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [30ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [32ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [33ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [33ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [51ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [52ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [54ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [55ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [131ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [133ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [135ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [135ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [528ms] refreshing cache from master, [528ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [531ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [532ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [536ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [537ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [907ms] refreshing cache from master, [907ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [910ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [913ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [915ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [916ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [3828ms] refreshing cache from master, [3829ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [3833ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@573717387(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@56ef25a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006) -> (continuation of Deferred@721822915 after retry RPC@381678170) -> (continuation of Deferred@338374110 after retry RPC@2112823331) -> (continuation of Deferred@377723853 after retry RPC@539314705), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b41cfc3 -> wakeup thread Time-limited test -> (continuation of Deferred@210085849 after retry RPC@698310237) -> (continuation of Deferred@1074292865 after retry RPC@1273676566) -> (continuation of Deferred@1840669317 after retry RPC@1884946006) -> (continuation of Deferred@721822915 after retry RPC@381678170) -> (continuation of Deferred@338374110 after retry RPC@2112823331) -> (continuation of Deferred@377723853 after retry RPC@539314705))) after lookup
17:13:58.960 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:13:58.963 [DEBUG - kudu-nio-0] (Connection.java:757) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] Successfully connected to peer
17:13:58.992 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xcd072c4b, L:/127.0.0.1:39380 - R:/127.6.166.67:35283] using TOKEN/null
17:13:59.042 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.65:45345
17:13:59.044 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

17:13:59.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:59.045363  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:59.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:13:59.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:59.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:13:59.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:13:59.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:45345
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35931
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:59.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:13:59.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.363715  8375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:13:59.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.364213  8375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:13:59.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.364705  8375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:13:59.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.396509  8375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:13:59.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.397377  8375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.65
17:13:59.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:59.432307  8375 tablet_server_runner.cc:78] Tablet server non-default flags:
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:45345
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35931
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:13:59.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:13:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:13:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:13:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:13:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:13:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:13:59.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:59.433866  8375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:13:59.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:59.435444  8375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:13:59.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.455928  8384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:13:59.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.542732  8022 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 -> Peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Couldn't send request to peer 5f827e4e0adc48ca87a385c561e3c660. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
17:13:59.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.972250  8132 debug-util.cc:398] Leaking SignalData structure 0x7b08000c4060 after lost signal to thread 8000
17:13:59.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.973377  8132 debug-util.cc:398] Leaking SignalData structure 0x7b08000d58e0 after lost signal to thread 8135
17:14:00.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.456038  8385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:00.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:13:59.457041  8375 server_base.cc:1034] running on GCE node
17:14:00.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:13:59.455940  8387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:00.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.711573  8375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:14:00.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.714560  8375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:14:00.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.715991  8375 hybrid_clock.cc:648] HybridClock initialized: now 1783530840715969 us; error 43 us; skew 500 ppm
17:14:00.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.716753  8375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:14:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.728536  8375 webserver.cc:460] Webserver started at http://127.6.166.65:35931/ using document root <none> and password file <none>
17:14:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.729936  8375 fs_manager.cc:359] Metadata directory not provided
17:14:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.730252  8375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:14:00.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.742005  8375 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.004s
17:14:00.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.747335  8398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:14:00.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.748553  8375 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
17:14:00.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.748906  8375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:28 on dist-test-slave-v72b"
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.751471  8375 fs_report.cc:383] FS layout report
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/data
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:14:00.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:14:00.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:14:00.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.794797  8375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:14:00.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.796402  8375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:14:00.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.796904  8375 kserver.cc:163] Server-wide thread pool size limit: 3276
17:14:00.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.800292  8375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:14:00.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.806692  8405 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:14:00.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.814978  8375 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:14:00.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.815253  8375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.001s
17:14:00.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.815677  8375 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:14:00.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.820577  8375 ts_tablet_manager.cc:610] Registered 1 tablets
17:14:00.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.820823  8375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
17:14:00.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.821182  8405 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap starting.
17:14:00.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:00.903160  8405 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Log is configured to *not* fsync() on all Append() calls
17:14:01.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.000315  8405 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:14:01.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.001338  8405 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap complete.
17:14:01.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.002760  8405 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent bootstrapping tablet: real 0.182s	user 0.105s	sys 0.065s
17:14:01.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.026949  8405 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:01.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.030671  8405 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Initialized, Role: FOLLOWER
17:14:01.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.031957  8405 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:01.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.055867  8405 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent starting tablet: real 0.053s	user 0.032s	sys 0.016s
17:14:01.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.138175  8375 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.65:45345
17:14:01.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.138352  8514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.65:45345 every 8 connection(s)
17:14:01.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.141358  8375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:14:01.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.150557  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 8375
17:14:01.152 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:01.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.219266  8516 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:14:01.222 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:14:01.222 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.67:35283
17:14:01.223 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

17:14:01.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.219751  8516 heartbeater.cc:464] Registering TS with master...
17:14:01.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.221024  8516 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:14:01.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.223712  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 7999
17:14:01.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.226622  6929 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:01.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.229875  8515 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:14:01.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.230276  8515 heartbeater.cc:464] Registering TS with master...
17:14:01.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.231338  8515 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:14:01.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.236228  7007 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:01.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.238688  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.65:36247
17:14:01.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.247804  8517 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:14:01.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.248204  8517 heartbeater.cc:464] Registering TS with master...
17:14:01.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.248982  8517 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:14:01.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:01.252899  6854 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:01.259 [DEBUG - kudu-nio-0] (Connection.java:239) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] handling channelInactive
17:14:01.259 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state READY due to: connection closed
17:14:01.260 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:01.260 [DEBUG - Time-limited test] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:01.262 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.262 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.262 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.275 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:01.275 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:01.276 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [14ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1116416626(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@ff63c74 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@75e8a7e3 -> wakeup thread Time-limited test)) after lookup
17:14:01.276 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:01.279 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.279 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.279 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.295 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:01.295 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:01.296 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [14ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [16ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [19ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [20ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1116416626(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@ff63c74 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@75e8a7e3 -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656))) after lookup
17:14:01.297 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:01.298 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.298 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.298 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:01.335 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:01.336 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [14ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [16ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [19ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [20ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [36ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [38ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [38ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [74ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1116416626(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@ff63c74 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@75e8a7e3 -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932))) after lookup
17:14:01.336 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:01.338 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.338 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.338 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.416 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:01.416 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:01.416 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=156), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [14ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [16ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [19ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [20ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [36ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [38ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [38ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [74ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [76ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [78ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [78ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [155ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1116416626(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@ff63c74 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@75e8a7e3 -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725))) after lookup
17:14:01.417 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:01.418 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.418 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.418 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.476 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:01.476 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:01.476 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=216), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [14ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [16ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [19ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [20ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [36ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [38ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [38ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [74ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [76ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [78ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [78ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [155ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [157ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [158ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [158ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [212ms] refreshing cache from master, [212ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [215ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1116416626(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@ff63c74 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725) -> (continuation of Deferred@1920968527 after retry RPC@579667604), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@75e8a7e3 -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725) -> (continuation of Deferred@1920968527 after retry RPC@579667604))) after lookup
17:14:01.477 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:01.479 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.479 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:01.479 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:02.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.185928  8526 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:14:02.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.186353  8526 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:02.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.188477  8526 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:02.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:02.193364  8401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:02.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:02.197227  8401 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:02.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.200750  8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" is_pre_election: true
17:14:02.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.202090  8400 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660; no voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72
17:14:02.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.202747  8526 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
17:14:02.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.243000  8515 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:14:02.675 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:02.675 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:02.676 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1415), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [14ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [16ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [19ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [20ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [36ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [38ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [38ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [74ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [76ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [78ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [78ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [155ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [157ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [158ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [158ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [212ms] refreshing cache from master, [212ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [215ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [217ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [219ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [219ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [1411ms] refreshing cache from master, [1411ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1415ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1116416626(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@ff63c74 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725) -> (continuation of Deferred@1920968527 after retry RPC@579667604) -> (continuation of Deferred@431475790 after retry RPC@1405403210), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@75e8a7e3 -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725) -> (continuation of Deferred@1920968527 after retry RPC@579667604) -> (continuation of Deferred@431475790 after retry RPC@1405403210))) after lookup
17:14:02.677 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:02.677 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:02.678 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:02.678 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:02.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.679174  8532 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dd20240671db45b885a8a5ca2364ec72)
17:14:02.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.679594  8532 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:02.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.681006  8532 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:02.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:02.683856  8205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:02.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:02.686997  8205 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:02.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.695020  8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660" is_pre_election: true
17:14:02.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.695816  8469 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 5.
17:14:02.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.696920  8204 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660; no voters: dd20240671db45b885a8a5ca2364ec72
17:14:02.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.697587  8532 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
17:14:02.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.697870  8532 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 6 FOLLOWER]: Starting leader election (detected failure of leader dd20240671db45b885a8a5ca2364ec72)
17:14:02.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.698088  8532 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 6 FOLLOWER]: Advancing to term 7
17:14:02.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.702538  8532 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:02.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.704033  8532 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 7 election: Requested vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:02.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.705035  8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:14:02.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.705564  8469 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 5 FOLLOWER]: Advancing to term 7
17:14:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:02.707046  8205 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:02.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.712430  8469 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 7.
17:14:02.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.713303  8204 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660; no voters: dd20240671db45b885a8a5ca2364ec72
17:14:02.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.714013  8532 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Leader election won for term 7
17:14:02.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.714865  8532 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 LEADER]: Becoming Leader. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Running, Role: LEADER
17:14:02.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.715726  8532 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:02.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:02.723762  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a reported cstate change: term changed from 6 to 7, leader changed from dd20240671db45b885a8a5ca2364ec72 (127.6.166.67) to 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66). New cstate: current_term: 7 leader_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: UNKNOWN } } }
17:14:03.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:03.160831  8205 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a -> Peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Couldn't send request to peer dd20240671db45b885a8a5ca2364ec72. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:14:03.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:03.335981  8469 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 7 FOLLOWER]: Refusing update from remote peer 13d4754fe07445a99ba61cb33e1d3b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
17:14:03.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:03.337239  8532 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
17:14:05.676 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:05.676 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:05.677 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4416), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [14ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [16ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [19ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [20ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [35ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [36ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [38ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [38ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [74ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [76ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [78ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [78ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [151ms] refreshing cache from master, [152ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [155ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [157ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [158ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [158ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [212ms] refreshing cache from master, [212ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [215ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [217ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [219ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [219ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [1411ms] refreshing cache from master, [1411ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1415ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [1416ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [1418ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [1418ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [4411ms] refreshing cache from master, [4412ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [4416ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1116416626(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@ff63c74 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725) -> (continuation of Deferred@1920968527 after retry RPC@579667604) -> (continuation of Deferred@431475790 after retry RPC@1405403210) -> (continuation of Deferred@1905007589 after retry RPC@1056230682), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@75e8a7e3 -> wakeup thread Time-limited test -> (continuation of Deferred@1573950150 after retry RPC@1832378656) -> (continuation of Deferred@881547459 after retry RPC@190475932) -> (continuation of Deferred@1365971026 after retry RPC@161327725) -> (continuation of Deferred@1920968527 after retry RPC@579667604) -> (continuation of Deferred@431475790 after retry RPC@1405403210) -> (continuation of Deferred@1905007589 after retry RPC@1056230682))) after lookup
17:14:05.678 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:05.679 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] Successfully connected to peer
17:14:05.698 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x1544d2d4, L:/127.0.0.1:56894 - R:/127.6.166.66:35735] using TOKEN/null
17:14:05.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:05.733584  8205 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a -> Peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Couldn't send request to peer dd20240671db45b885a8a5ca2364ec72. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
17:14:05.755 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.67:35283
17:14:05.755 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:05.757177  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:05.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:35283
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44511
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:14:05.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:05.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:05.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:05.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:14:06.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.081578  8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:14:06.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.082041  8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:14:06.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.082602  8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:14:06.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.116050  8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:14:06.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.116837  8572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.67
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:06.150751  8572 tablet_server_runner.cc:78] Tablet server non-default flags:
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:35283
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44511
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:14:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:14:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:14:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:14:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:14:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:14:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:14:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:06.151880  8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:14:06.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:06.153401  8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:14:06.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.172668  8582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:06.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.562673  8511 debug-util.cc:398] Leaking SignalData structure 0x7b08000c42c0 after lost signal to thread 8379
17:14:06.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.564345  8511 debug-util.cc:398] Leaking SignalData structure 0x7b080009e140 after lost signal to thread 8514
17:14:07.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.172703  8580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:07.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:06.172883  8579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:07.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:06.173913  8572 server_base.cc:1034] running on GCE node
17:14:07.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.430745  8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:14:07.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.433741  8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:14:07.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.435276  8572 hybrid_clock.cc:648] HybridClock initialized: now 1783530847435208 us; error 43 us; skew 500 ppm
17:14:07.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.436376  8572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:14:07.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.445607  8572 webserver.cc:460] Webserver started at http://127.6.166.67:44511/ using document root <none> and password file <none>
17:14:07.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.446895  8572 fs_manager.cc:359] Metadata directory not provided
17:14:07.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.447177  8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:14:07.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.458658  8572 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.002s	sys 0.004s
17:14:07.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.464555  8589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:14:07.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.466014  8572 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.005s
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.466431  8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dd20240671db45b885a8a5ca2364ec72"
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:33 on dist-test-slave-v72b"
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.470017  8572 fs_report.cc:383] FS layout report
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/data
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:14:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:14:07.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:14:07.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:14:07.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:14:07.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.545254  8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:14:07.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.547294  8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:14:07.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.547875  8572 kserver.cc:163] Server-wide thread pool size limit: 3276
17:14:07.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.551589  8572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:14:07.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.559656  8596 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:14:07.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.568059  8572 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:14:07.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.568387  8572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
17:14:07.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.568717  8572 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:14:07.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.576905  8572 ts_tablet_manager.cc:610] Registered 1 tablets
17:14:07.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.577179  8572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s	user 0.007s	sys 0.001s
17:14:07.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.577570  8596 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap starting.
17:14:07.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.656255  8596 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Log is configured to *not* fsync() on all Append() calls
17:14:07.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.762235  8596 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:14:07.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.763549  8596 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap complete.
17:14:07.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.765434  8596 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent bootstrapping tablet: real 0.188s	user 0.143s	sys 0.032s
17:14:07.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.790869  8596 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:07.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.794713  8596 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd20240671db45b885a8a5ca2364ec72, State: Initialized, Role: FOLLOWER
17:14:07.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.795974  8596 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.800714  8596 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent starting tablet: real 0.035s	user 0.024s	sys 0.008s
17:14:07.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.816906  8572 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.67:35283
17:14:07.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.817028  8704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.67:35283 every 8 connection(s)
17:14:07.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.820678  8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:14:07.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.825718  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 8572
17:14:07.827 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:07.862 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:14:07.864 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.66:35735
17:14:07.864 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:14:07.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.864679  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 8181
17:14:07.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.887554  8706 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:14:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.888078  8706 heartbeater.cc:464] Registering TS with master...
17:14:07.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.889312  8706 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:14:07.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.894711  6929 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:07.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.898375  8705 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:14:07.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.898860  8705 heartbeater.cc:464] Registering TS with master...
17:14:07.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.900017  8705 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:14:07.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.903574  7007 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:07.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.908308  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.67:50935
17:14:07.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.917464  8707 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:14:07.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.917866  8707 heartbeater.cc:464] Registering TS with master...
17:14:07.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.918632  8707 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:14:07.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:07.922356  6854 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:07.937 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] handling channelInactive
17:14:07.937 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state READY due to: connection closed
17:14:07.939 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:07.939 [DEBUG - Time-limited test] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:07.944 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.944 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.945 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.955 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:07.956 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:07.956 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [6ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [6ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1945752008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2a08bed5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@716eb57e -> wakeup thread Time-limited test)) after lookup
17:14:07.957 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:07.959 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.959 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.959 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:07.976 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:07.976 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [6ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [6ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [18ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [20ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [20ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [36ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1945752008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2a08bed5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@716eb57e -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967))) after lookup
17:14:07.977 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:07.978 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.978 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.978 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.994 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:07.995 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:07.995 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [6ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [6ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [18ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [20ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [20ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [36ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [37ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [39ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [39ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [55ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1945752008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2a08bed5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@716eb57e -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285))) after lookup
17:14:07.996 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:07.998 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.998 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:07.998 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:08.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:08.116 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:08.116 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=177), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [6ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [6ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [18ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [20ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [20ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [36ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [37ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [39ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [39ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [55ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [57ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [59ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [59ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [172ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [176ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1945752008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2a08bed5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285) -> (continuation of Deferred@816658647 after retry RPC@1196079329), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@716eb57e -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285) -> (continuation of Deferred@816658647 after retry RPC@1196079329))) after lookup
17:14:08.117 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:08.118 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:08.118 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:08.118 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:08.535 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:08.535 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:08.536 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=596), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [6ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [6ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [18ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [20ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [20ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [36ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [37ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [39ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [39ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [55ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [57ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [59ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [59ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [172ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [176ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [177ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [179ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [179ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [592ms] refreshing cache from master, [592ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [596ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1945752008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2a08bed5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285) -> (continuation of Deferred@816658647 after retry RPC@1196079329) -> (continuation of Deferred@782651891 after retry RPC@1928393136), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@716eb57e -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285) -> (continuation of Deferred@816658647 after retry RPC@1196079329) -> (continuation of Deferred@782651891 after retry RPC@1928393136))) after lookup
17:14:08.537 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:08.538 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:08.538 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:08.539 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:08.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:08.913110  8705 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:14:09.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.245622  8717 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:14:09.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.246105  8717 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:09.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.248301  8717 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:09.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:09.253032  8591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:14:09.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:09.257329  8591 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:14:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.263870  8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660" is_pre_election: true
17:14:09.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.265378  8591 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660
17:14:09.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.266122  8717 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
17:14:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.277262  8722 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:14:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.277705  8722 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:09.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.279446  8722 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:09.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:09.281385  8400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:14:09.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:09.295125  8400 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:14:09.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.303977  8659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:14:09.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.304641  8659 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 6.
17:14:09.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.305703  8401 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:14:09.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.306339  8722 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 7 FOLLOWER]: Leader pre-election won for term 8
17:14:09.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.306653  8722 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:14:09.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.306960  8722 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 7 FOLLOWER]: Advancing to term 8
17:14:09.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.311417  8722 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:09.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.312834  8722 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 8 election: Requested vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:09.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.313925  8659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72"
17:14:09.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.314554  8659 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 6 FOLLOWER]: Advancing to term 8
17:14:09.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:09.316556  8400 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:14:09.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.321491  8659 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 8.
17:14:09.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.322247  8401 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:14:09.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.322790  8722 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Leader election won for term 8
17:14:09.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.323858  8722 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 LEADER]: Becoming Leader. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Running, Role: LEADER
17:14:09.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.324719  8722 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:09.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.333907  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 reported cstate change: term changed from 7 to 8, leader changed from 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66) to 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65). New cstate: current_term: 8 leader_uuid: "5f827e4e0adc48ca87a385c561e3c660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: HEALTHY } } }
17:14:09.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.724442  8659 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 8 FOLLOWER]: Refusing update from remote peer 5f827e4e0adc48ca87a385c561e3c660: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
17:14:09.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.725951  8722 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.001s
17:14:09.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:09.771476  8400 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 -> Peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Couldn't send request to peer 13d4754fe07445a99ba61cb33e1d3b2a. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:14:09.855 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:09.855 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:09.856 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1916), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [6ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [6ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [16ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [18ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [20ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [20ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [36ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [37ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [39ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [39ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [55ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [57ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [59ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [59ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [172ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [176ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [177ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [179ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [179ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [592ms] refreshing cache from master, [592ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [596ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [597ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [599ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [600ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [1912ms] refreshing cache from master, [1912ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1915ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1945752008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@2a08bed5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285) -> (continuation of Deferred@816658647 after retry RPC@1196079329) -> (continuation of Deferred@782651891 after retry RPC@1928393136) -> (continuation of Deferred@88415517 after retry RPC@1892483444), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@716eb57e -> wakeup thread Time-limited test -> (continuation of Deferred@1143157056 after retry RPC@1080211967) -> (continuation of Deferred@1154974694 after retry RPC@1516613285) -> (continuation of Deferred@816658647 after retry RPC@1196079329) -> (continuation of Deferred@782651891 after retry RPC@1928393136) -> (continuation of Deferred@88415517 after retry RPC@1892483444))) after lookup
17:14:09.856 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:09.857 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] Successfully connected to peer
17:14:09.879 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xeefa8041, L:/127.0.0.1:57092 - R:/127.6.166.65:45345] using TOKEN/null
17:14:09.941 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.66:35735
17:14:09.942 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:14:09.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:09.943519  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:09.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:14:10.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.247876  8751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:14:10.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.248442  8751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:14:10.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.249046  8751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:14:10.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.297930  8751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:14:10.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.299306  8751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.66
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:10.343771  8751 tablet_server_runner.cc:78] Tablet server non-default flags:
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:14:10.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:14:10.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:14:10.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:14:10.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:14:10.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:10.344863  8751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:14:10.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:10.346352  8751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:14:10.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.364887  8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:11.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.364959  8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:11.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:10.367635  8751 server_base.cc:1034] running on GCE node
17:14:11.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:10.365895  8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:11.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.574878  8751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:14:11.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.577333  8751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:14:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.578747  8751 hybrid_clock.cc:648] HybridClock initialized: now 1783530851578716 us; error 43 us; skew 500 ppm
17:14:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.579723  8751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:14:11.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.586532  8751 webserver.cc:460] Webserver started at http://127.6.166.66:40641/ using document root <none> and password file <none>
17:14:11.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.587702  8751 fs_manager.cc:359] Metadata directory not provided
17:14:11.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.587905  8751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:14:11.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.596601  8751 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.004s
17:14:11.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.601236  8769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:14:11.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.602335  8751 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.602697  8751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:31 on dist-test-slave-v72b"
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.605054  8751 fs_report.cc:383] FS layout report
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/data
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:14:11.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:14:11.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:14:11.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:14:11.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:14:11.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:14:11.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:14:11.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.649708  8751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:14:11.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.651363  8751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:14:11.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.651866  8751 kserver.cc:163] Server-wide thread pool size limit: 3276
17:14:11.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.654637  8751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:14:11.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.660678  8776 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:14:11.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.668355  8751 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:14:11.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.668648  8751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
17:14:11.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.669065  8751 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:14:11.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.675684  8751 ts_tablet_manager.cc:610] Registered 1 tablets
17:14:11.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.675933  8751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s	user 0.007s	sys 0.000s
17:14:11.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.676313  8776 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap starting.
17:14:11.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.743499  8776 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Log is configured to *not* fsync() on all Append() calls
17:14:11.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.861100  8776 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:14:11.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.861964  8776 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap complete.
17:14:11.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.863386  8776 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent bootstrapping tablet: real 0.188s	user 0.134s	sys 0.045s
17:14:11.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.896243  8776 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:11.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.899935  8776 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Initialized, Role: FOLLOWER
17:14:11.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.900959  8776 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:11.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.921572  8776 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent starting tablet: real 0.058s	user 0.051s	sys 0.004s
17:14:11.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.924144  8884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.66:35735 every 8 connection(s)
17:14:11.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.924700  8751 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.66:35735
17:14:11.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.929466  8751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:14:11.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.938344  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 8751
17:14:11.940 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:11.977 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:14:11.983 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.65:45345
17:14:11.983 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

17:14:11.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.983865  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 8375
17:14:12.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.999071  8885 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:14:12.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:11.999720  8885 heartbeater.cc:464] Registering TS with master...
17:14:12.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.001418  8885 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:14:12.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.005252  8886 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:14:12.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.005640  8886 heartbeater.cc:464] Registering TS with master...
17:14:12.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.005895  7007 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:14:12.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.006557  8886 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:14:12.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.008845  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.66:44843
17:14:12.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.010453  6929 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:14:12.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.022678  8887 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:14:12.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.023084  8887 heartbeater.cc:464] Registering TS with master...
17:14:12.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.023775  8887 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:14:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:12.026829  6854 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:14:12.035 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] handling channelInactive
17:14:12.036 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state READY due to: connection closed
17:14:12.036 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:12.037 [DEBUG - Time-limited test] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:12.038 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.039 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.039 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.056 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:12.056 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:12.056 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [3ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [3ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1036467256(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@19fcfe6 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4cfa1958 -> wakeup thread Time-limited test)) after lookup
17:14:12.057 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:12.058 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.058 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.058 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.075 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:12.076 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:12.076 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [3ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [3ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [20ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [22ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [22ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1036467256(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@19fcfe6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4cfa1958 -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915))) after lookup
17:14:12.076 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:12.078 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.078 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.078 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:12.115 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:12.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [3ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [3ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [20ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [22ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [22ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [42ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [42ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1036467256(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@19fcfe6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4cfa1958 -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135))) after lookup
17:14:12.116 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:12.117 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.117 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.117 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.216 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:12.216 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:12.216 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=179), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [3ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [3ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [20ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [22ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [22ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [42ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [42ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [80ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [81ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [81ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [179ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1036467256(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@19fcfe6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135) -> (continuation of Deferred@188777333 after retry RPC@759646624), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4cfa1958 -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135) -> (continuation of Deferred@188777333 after retry RPC@759646624))) after lookup
17:14:12.217 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:12.218 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.219 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.219 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.676 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:12.676 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:12.677 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=640), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [3ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [3ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [20ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [22ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [22ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [42ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [42ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [80ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [81ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [81ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [179ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [180ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [183ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [183ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [635ms] refreshing cache from master, [635ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [639ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1036467256(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@19fcfe6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135) -> (continuation of Deferred@188777333 after retry RPC@759646624) -> (continuation of Deferred@1908104369 after retry RPC@1713770863), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4cfa1958 -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135) -> (continuation of Deferred@188777333 after retry RPC@759646624) -> (continuation of Deferred@1908104369 after retry RPC@1713770863))) after lookup
17:14:12.677 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:12.678 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.678 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:12.678 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345
17:14:13.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.014727  8885 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:14:13.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.156800  8897 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:14:13.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.157197  8897 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:13.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.159596  8897 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:13.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:13.163031  8771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:14:13.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:13.167416  8771 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:14:13.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.172045  8659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:14:13.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.173621  8772 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a; no voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72
17:14:13.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.174394  8897 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
17:14:13.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.322830  8902 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5f827e4e0adc48ca87a385c561e3c660)
17:14:13.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.323309  8902 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:13.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.325017  8902 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:13.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:13.331629  8591 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:14:13.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.337399  8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" is_pre_election: true
17:14:13.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.338198  8839 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd20240671db45b885a8a5ca2364ec72 in term 7.
17:14:13.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.339409  8591 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 5f827e4e0adc48ca87a385c561e3c660
17:14:13.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.340116  8902 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 8 FOLLOWER]: Leader pre-election won for term 9
17:14:13.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.340421  8902 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5f827e4e0adc48ca87a385c561e3c660)
17:14:13.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.340656  8902 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 8 FOLLOWER]: Advancing to term 9
17:14:13.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.344805  8902 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:13.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.346078  8902 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 9 election: Requested vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.346853  8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "dd20240671db45b885a8a5ca2364ec72" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:14:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.347337  8839 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 7 FOLLOWER]: Advancing to term 9
17:14:13.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:13.349133  8591 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111)
17:14:13.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.354089  8839 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd20240671db45b885a8a5ca2364ec72 in term 9.
17:14:13.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.355024  8591 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72; no voters: 5f827e4e0adc48ca87a385c561e3c660
17:14:13.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.355722  8902 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 9 FOLLOWER]: Leader election won for term 9
17:14:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.356688  8902 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 9 LEADER]: Becoming Leader. State: Replica: dd20240671db45b885a8a5ca2364ec72, State: Running, Role: LEADER
17:14:13.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.357458  8902 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:13.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.365556  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 reported cstate change: term changed from 8 to 9, leader changed from 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65) to dd20240671db45b885a8a5ca2364ec72 (127.6.166.67). New cstate: current_term: 9 leader_uuid: "dd20240671db45b885a8a5ca2364ec72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: UNKNOWN } } }
17:14:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.742462  8839 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 9 FOLLOWER]: Refusing update from remote peer dd20240671db45b885a8a5ca2364ec72: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
17:14:13.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:13.744041  8902 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
17:14:13.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:13.785388  8591 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 -> Peer 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345): Couldn't send request to peer 5f827e4e0adc48ca87a385c561e3c660. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:14:13.995 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:13.995 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:13.996 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1959), Traces: [0ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [3ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [3ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [19ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [20ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [22ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [22ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [39ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [42ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [42ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [80ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [81ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [81ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [179ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [180ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [183ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [183ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [635ms] refreshing cache from master, [635ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [639ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [641ms] sending RPC to server 5f827e4e0adc48ca87a385c561e3c660, [642ms] received response from server 5f827e4e0adc48ca87a385c561e3c660: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [642ms] delaying RPC due to: Network error: Failed to connect to peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345): Connection refused: /127.6.166.65:45345, [1955ms] refreshing cache from master, [1955ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1958ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1036467256(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@19fcfe6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135) -> (continuation of Deferred@188777333 after retry RPC@759646624) -> (continuation of Deferred@1908104369 after retry RPC@1713770863) -> (continuation of Deferred@29399151 after retry RPC@1436709638), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4cfa1958 -> wakeup thread Time-limited test -> (continuation of Deferred@1494025840 after retry RPC@965476915) -> (continuation of Deferred@1940854395 after retry RPC@720052135) -> (continuation of Deferred@188777333 after retry RPC@759646624) -> (continuation of Deferred@1908104369 after retry RPC@1713770863) -> (continuation of Deferred@29399151 after retry RPC@1436709638))) after lookup
17:14:13.997 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:13.998 [DEBUG - kudu-nio-0] (Connection.java:757) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] Successfully connected to peer
17:14:14.017 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x788bc50c, L:/127.0.0.1:48386 - R:/127.6.166.67:35283] using TOKEN/null
17:14:14.058 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.65:45345
17:14:14.058 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

17:14:14.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:14.059602  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:14.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:45345
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35931
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:14:14.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.363651  8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:14:14.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.364114  8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:14:14.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.364687  8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:14:14.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.397784  8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:14:14.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.398639  8930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.65
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:14.433175  8930 tablet_server_runner.cc:78] Tablet server non-default flags:
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.65:45345
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:14:14.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.65
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35931
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.65
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/logs
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:14:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:14.434296  8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:14:14.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:14.435942  8930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:14:14.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.455747  8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:14.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.670112  8591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.65:45345: connect: Connection refused (error 111) [suppressed 4 similar messages]
17:14:15.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:14.457906  8930 server_base.cc:1034] running on GCE node
17:14:15.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.456612  8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:15.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:14.456403  8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:15.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.663748  8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:14:15.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.666090  8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:14:15.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.667416  8930 hybrid_clock.cc:648] HybridClock initialized: now 1783530855667384 us; error 48 us; skew 500 ppm
17:14:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.668550  8930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:14:15.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.675796  8930 webserver.cc:460] Webserver started at http://127.6.166.65:35931/ using document root <none> and password file <none>
17:14:15.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.676669  8930 fs_manager.cc:359] Metadata directory not provided
17:14:15.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.676854  8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:14:15.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.684522  8930 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.001s
17:14:15.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.688845  8953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:14:15.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.689872  8930 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
17:14:15.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.690228  8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:15.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:14:15.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:28 on dist-test-slave-v72b"
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.692795  8930 fs_report.cc:383] FS layout report
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/wal
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/data
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:14:15.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:14:15.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.735574  8930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:14:15.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.737475  8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:14:15.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.738018  8930 kserver.cc:163] Server-wide thread pool size limit: 3276
17:14:15.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.741240  8930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:14:15.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.748729  8960 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:14:15.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.757651  8930 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:14:15.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.757957  8930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
17:14:15.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.758253  8930 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:14:15.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.764163  8930 ts_tablet_manager.cc:610] Registered 1 tablets
17:14:15.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.764338  8930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.005s	sys 0.001s
17:14:15.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.764846  8960 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap starting.
17:14:15.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.842550  8960 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Log is configured to *not* fsync() on all Append() calls
17:14:15.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.969339  8930 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.65:45345
17:14:15.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.969522  9067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.65:45345 every 8 connection(s)
17:14:15.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.972862  8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-0/data/info.pb
17:14:15.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.975936  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 8930
17:14:15.976 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:15.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.984782  8960 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:14:15.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.985954  8960 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Bootstrap complete.
17:14:15.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:15.988180  8960 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent bootstrapping tablet: real 0.224s	user 0.164s	sys 0.047s
17:14:16.022 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:14:16.023 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.67:35283
17:14:16.023 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

17:14:16.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.024170  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 8572
17:14:16.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.028040  8960 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:16.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.032675  8960 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Initialized, Role: FOLLOWER
17:14:16.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.033891  8960 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:16.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.052902  8960 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660: Time spent starting tablet: real 0.064s	user 0.031s	sys 0.005s
17:14:16.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.057267  9069 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:14:16.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.057648  9069 heartbeater.cc:464] Registering TS with master...
17:14:16.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.058777  9069 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:14:16.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.060988  9068 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:14:16.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.061312  9068 heartbeater.cc:464] Registering TS with master...
17:14:16.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.062129  9068 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:14:16.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.063514  6928 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:16.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.065486  7006 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:16.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.069576  7006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.65:44467
17:14:16.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.073320  9068 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:14:16.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.075803  9070 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:14:16.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.076176  9070 heartbeater.cc:464] Registering TS with master...
17:14:16.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.076920  9070 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:14:16.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:16.079948  6854 ts_manager.cc:194] Re-registered known tserver with Master: 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:16.085 [DEBUG - kudu-nio-0] (Connection.java:239) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] handling channelInactive
17:14:16.085 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:16.085 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state READY due to: connection closed
17:14:16.086 [DEBUG - Time-limited test] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:16.088 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.088 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.088 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:16.095 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:16.095 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [8ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1058977258(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@1dcc933c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@deb8e68 -> wakeup thread Time-limited test)) after lookup
17:14:16.096 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:16.097 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.097 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.097 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:16.115 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:16.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [8ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [9ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [11ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [28ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1058977258(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@1dcc933c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@deb8e68 -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411))) after lookup
17:14:16.116 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:16.117 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.118 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.118 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.155 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:16.155 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:16.156 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [8ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [9ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [11ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [28ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [30ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [32ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [32ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [68ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1058977258(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@1dcc933c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@deb8e68 -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249))) after lookup
17:14:16.156 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:16.158 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.158 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.158 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.214 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:16.215 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:16.215 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=129), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [8ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [9ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [11ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [28ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [30ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [32ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [32ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [68ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [70ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [72ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [72ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [128ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1058977258(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@1dcc933c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249) -> (continuation of Deferred@1324548102 after retry RPC@1540048722), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@deb8e68 -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249) -> (continuation of Deferred@1324548102 after retry RPC@1540048722))) after lookup
17:14:16.216 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:16.217 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.217 [DEBUG - kudu-nio-1] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.217 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.256 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:16.256 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)[L]]}]
17:14:16.256 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=170), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [8ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [9ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [11ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [28ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [30ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [32ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [32ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [68ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [70ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [72ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [72ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [128ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [129ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [131ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [131ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [165ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [169ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1058977258(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@1dcc933c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249) -> (continuation of Deferred@1324548102 after retry RPC@1540048722) -> (continuation of Deferred@1829439631 after retry RPC@402913895), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@deb8e68 -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249) -> (continuation of Deferred@1324548102 after retry RPC@1540048722) -> (continuation of Deferred@1829439631 after retry RPC@402913895))) after lookup
17:14:16.257 [DEBUG - kudu-nio-1] (Connection.java:748) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] connecting to peer
17:14:16.259 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.259 [DEBUG - kudu-nio-0] (Connection.java:709) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] cleaning up while in state CONNECTING due to: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:16.259 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283
17:14:17.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.189620  9080 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
17:14:17.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.190222  9080 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.192726  9080 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:17.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.197412  8956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:17.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.201259  8956 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:17.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.204955  8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" is_pre_election: true
17:14:17.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.206284  8955 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660; no voters: 13d4754fe07445a99ba61cb33e1d3b2a, dd20240671db45b885a8a5ca2364ec72
17:14:17.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.207015  9080 raft_consensus.cc:2726] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
17:14:17.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.264213  9085 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader dd20240671db45b885a8a5ca2364ec72)
17:14:17.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.264681  9085 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:17.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.266494  9085 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:17.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.287204  8772 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:17.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.289218  9023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660" is_pre_election: true
17:14:17.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.289866  9023 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 8.
17:14:17.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.291030  8771 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660; no voters: dd20240671db45b885a8a5ca2364ec72
17:14:17.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.291821  9085 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
17:14:17.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.292073  9085 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader dd20240671db45b885a8a5ca2364ec72)
17:14:17.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.292361  9085 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 9 FOLLOWER]: Advancing to term 10
17:14:17.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.296623  9085 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:17.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.298009  9085 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 10 election: Requested vote from peers dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283), 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65:45345)
17:14:17.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.299080  9023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "5f827e4e0adc48ca87a385c561e3c660"
17:14:17.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.299708  9023 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 8 FOLLOWER]: Advancing to term 10
17:14:17.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.301532  8772 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111)
17:14:17.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.306708  9023 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d4754fe07445a99ba61cb33e1d3b2a in term 10.
17:14:17.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.307772  8771 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d4754fe07445a99ba61cb33e1d3b2a, 5f827e4e0adc48ca87a385c561e3c660; no voters: dd20240671db45b885a8a5ca2364ec72
17:14:17.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.308457  9085 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 10 FOLLOWER]: Leader election won for term 10
17:14:17.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.309556  9085 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 10 LEADER]: Becoming Leader. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Running, Role: LEADER
17:14:17.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.310490  9085 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:17.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.320823  7006 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a reported cstate change: term changed from 9 to 10, leader changed from dd20240671db45b885a8a5ca2364ec72 (127.6.166.67) to 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66). New cstate: current_term: 10 leader_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: UNKNOWN } } }
17:14:17.535 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:17.535 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:17.536 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1450), Traces: [0ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [2ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [2ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [8ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [9ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [11ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [11ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [28ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [30ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [32ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [32ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [68ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [70ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [72ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [72ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [128ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [129ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [131ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [131ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [165ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [169ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [171ms] sending RPC to server dd20240671db45b885a8a5ca2364ec72, [173ms] received response from server dd20240671db45b885a8a5ca2364ec72: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [173ms] delaying RPC due to: Network error: Failed to connect to peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283): Connection refused: /127.6.166.67:35283, [1445ms] refreshing cache from master, [1446ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1449ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1058977258(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@1dcc933c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249) -> (continuation of Deferred@1324548102 after retry RPC@1540048722) -> (continuation of Deferred@1829439631 after retry RPC@402913895) -> (continuation of Deferred@761316122 after retry RPC@716690451), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@deb8e68 -> wakeup thread Time-limited test -> (continuation of Deferred@189802063 after retry RPC@1520001411) -> (continuation of Deferred@1003286547 after retry RPC@611728249) -> (continuation of Deferred@1324548102 after retry RPC@1540048722) -> (continuation of Deferred@1829439631 after retry RPC@402913895) -> (continuation of Deferred@761316122 after retry RPC@716690451))) after lookup
17:14:17.536 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:17.537 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] Successfully connected to peer
17:14:17.554 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x075981e7, L:/127.0.0.1:55742 - R:/127.6.166.66:35735] using TOKEN/null
17:14:17.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.562687  9023 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 10 FOLLOWER]: Refusing update from remote peer 13d4754fe07445a99ba61cb33e1d3b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
17:14:17.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.564311  9090 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.001s
17:14:17.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.565024  8772 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a -> Peer dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283): Couldn't send request to peer dd20240671db45b885a8a5ca2364ec72. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:14:17.623 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.67:35283
17:14:17.624 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

17:14:17.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:17.625419  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:17.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:17.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:17.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:14:17.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:35283
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44511
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:14:17.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.934790  9110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:14:17.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.935312  9110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:14:17.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.935772  9110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:14:17.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.968897  9110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:14:17.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:17.969847  9110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.67
17:14:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:18.005033  9110 tablet_server_runner.cc:78] Tablet server non-default flags:
17:14:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:18.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.67:35283
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.67
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44511
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.67
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/logs
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:14:18.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:18.006526  9110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:14:18.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:18.008198  9110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:14:18.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:18.027498  9119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:18.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:18.450476  8772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.67:35283: connect: Connection refused (error 111) [suppressed 4 similar messages]
17:14:19.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:18.027611  9117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:19.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:18.027498  9116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:19.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:18.029428  9110 server_base.cc:1034] running on GCE node
17:14:19.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.250414  9110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:14:19.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.252736  9110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:14:19.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.254099  9110 hybrid_clock.cc:648] HybridClock initialized: now 1783530859254064 us; error 47 us; skew 500 ppm
17:14:19.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.255033  9110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:14:19.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.266747  9110 webserver.cc:460] Webserver started at http://127.6.166.67:44511/ using document root <none> and password file <none>
17:14:19.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.268095  9110 fs_manager.cc:359] Metadata directory not provided
17:14:19.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.268371  9110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:14:19.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.277086  9110 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
17:14:19.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.281838  9130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:14:19.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.283051  9110 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
17:14:19.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.283422  9110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:19.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "dd20240671db45b885a8a5ca2364ec72"
17:14:19.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:33 on dist-test-slave-v72b"
17:14:19.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.285938  9110 fs_report.cc:383] FS layout report
17:14:19.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:14:19.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/wal
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/data
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:14:19.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:14:19.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.326426  9110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:14:19.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.328099  9110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:14:19.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.328627  9110 kserver.cc:163] Server-wide thread pool size limit: 3276
17:14:19.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.331434  9110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:14:19.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.337693  9137 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:14:19.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.345265  9110 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:14:19.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.345530  9110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s	user 0.002s	sys 0.001s
17:14:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.345880  9110 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:14:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.350997  9110 ts_tablet_manager.cc:610] Registered 1 tablets
17:14:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.351203  9110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
17:14:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.351564  9137 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap starting.
17:14:19.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.420792  9137 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Log is configured to *not* fsync() on all Append() calls
17:14:19.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.527593  9137 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:14:19.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.528396  9137 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Bootstrap complete.
17:14:19.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.529789  9137 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent bootstrapping tablet: real 0.179s	user 0.142s	sys 0.032s
17:14:19.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.534998  9110 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.67:35283
17:14:19.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.535271  9245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.67:35283 every 8 connection(s)
17:14:19.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.537504  9110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-2/data/info.pb
17:14:19.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.540161  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 9110
17:14:19.540 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:19.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.556257  9137 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:19.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.559867  9137 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd20240671db45b885a8a5ca2364ec72, State: Initialized, Role: FOLLOWER
17:14:19.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.561728  9137 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:19.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.573748  9137 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72: Time spent starting tablet: real 0.044s	user 0.032s	sys 0.007s
17:14:19.593 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:14:19.595 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.6.166.66:35735
17:14:19.596 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:14:19.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.596561  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 8751
17:14:19.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.605017  9247 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:14:19.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.605499  9247 heartbeater.cc:464] Registering TS with master...
17:14:19.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.606572  9247 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:14:19.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.607187  9248 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:14:19.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.607545  9248 heartbeater.cc:464] Registering TS with master...
17:14:19.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.608388  9248 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:14:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.610944  6928 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:19.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.614637  6853 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.620671  9246 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:14:19.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.621043  9246 heartbeater.cc:464] Registering TS with master...
17:14:19.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.621863  9246 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:14:19.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.624756  7007 ts_manager.cc:194] Re-registered known tserver with Master: dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.628185  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.67:38001
17:14:19.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:19.631596  9246 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:14:19.635 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] handling channelInactive
17:14:19.635 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state READY due to: connection closed
17:14:19.636 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:19.637 [DEBUG - Time-limited test] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:19.638 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.638 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.638 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.654 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:19.655 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:19.655 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [2ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [18ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1922799331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@20b78537 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cb07539 -> wakeup thread Time-limited test)) after lookup
17:14:19.656 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:19.656 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.657 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.657 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.675 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:19.675 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:19.676 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [2ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [18ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [19ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [21ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [21ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1922799331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@20b78537 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cb07539 -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046))) after lookup
17:14:19.676 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:19.677 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.677 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.677 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.715 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:19.715 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:19.715 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [2ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [18ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [19ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [21ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [21ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [41ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [41ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1922799331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@20b78537 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cb07539 -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970))) after lookup
17:14:19.716 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:19.717 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.717 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.717 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:19.835 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:19.836 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=199), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [2ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [18ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [19ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [21ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [21ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [41ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [41ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [80ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [81ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [81ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [195ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [199ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1922799331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@20b78537 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cb07539 -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305))) after lookup
17:14:19.837 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:19.838 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.838 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.838 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.854 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:19.854 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:19.855 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=218), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [2ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [18ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [19ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [21ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [21ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [41ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [41ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [80ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [81ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [81ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [195ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [199ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [200ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [202ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [202ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [215ms] refreshing cache from master, [215ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [218ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1922799331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@20b78537 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305) -> (continuation of Deferred@1899505494 after retry RPC@662905036), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cb07539 -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305) -> (continuation of Deferred@1899505494 after retry RPC@662905036))) after lookup
17:14:19.856 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:19.856 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.857 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:19.857 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:20.736 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:20.737 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)[L],5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345),dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:20.737 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1101), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [2ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [18ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [19ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [21ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [21ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [41ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [41ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [80ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [81ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [81ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [195ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [199ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [200ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [202ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [202ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [215ms] refreshing cache from master, [215ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [218ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [219ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [221ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [221ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [1095ms] refreshing cache from master, [1095ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1100ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1922799331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@20b78537 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305) -> (continuation of Deferred@1899505494 after retry RPC@662905036) -> (continuation of Deferred@1697088421 after retry RPC@1646068), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cb07539 -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305) -> (continuation of Deferred@1899505494 after retry RPC@662905036) -> (continuation of Deferred@1697088421 after retry RPC@1646068))) after lookup
17:14:20.738 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] connecting to peer
17:14:20.739 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:20.739 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] cleaning up while in state CONNECTING due to: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:20.739 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735) for tablet e6a7a34859d84fae9805e99114ca43db: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735
17:14:20.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.744369  9259 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:14:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.744829  9259 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.746608  9259 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:20.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:20.765108  8955 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:14:20.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.772078  9201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72" is_pre_election: true
17:14:20.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.772917  9201 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 9.
17:14:20.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.774106  8956 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:14:20.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.774711  9259 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 10 FOLLOWER]: Leader pre-election won for term 11
17:14:20.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.775040  9259 raft_consensus.cc:482] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 13d4754fe07445a99ba61cb33e1d3b2a)
17:14:20.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.775445  9259 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 10 FOLLOWER]: Advancing to term 11
17:14:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.780810  9259 raft_consensus.cc:504] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:20.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.782526  9259 leader_election.cc:288] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 11 election: Requested vote from peers 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735), dd20240671db45b885a8a5ca2364ec72 (127.6.166.67:35283)
17:14:20.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.783747  9201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a7a34859d84fae9805e99114ca43db" candidate_uuid: "5f827e4e0adc48ca87a385c561e3c660" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "dd20240671db45b885a8a5ca2364ec72"
17:14:20.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.784343  9201 raft_consensus.cc:3037] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 9 FOLLOWER]: Advancing to term 11
17:14:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:20.785859  8955 leader_election.cc:334] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111)
17:14:20.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.791702  9201 raft_consensus.cc:2454] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f827e4e0adc48ca87a385c561e3c660 in term 11.
17:14:20.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.792647  8956 leader_election.cc:302] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f827e4e0adc48ca87a385c561e3c660, dd20240671db45b885a8a5ca2364ec72; no voters: 13d4754fe07445a99ba61cb33e1d3b2a
17:14:20.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.793249  9259 raft_consensus.cc:2781] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 11 FOLLOWER]: Leader election won for term 11
17:14:20.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.794368  9259 raft_consensus.cc:686] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [term 11 LEADER]: Becoming Leader. State: Replica: 5f827e4e0adc48ca87a385c561e3c660, State: Running, Role: LEADER
17:14:20.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.795190  9259 consensus_queue.cc:237] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:20.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:20.802983  7007 catalog_manager.cc:5495] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 reported cstate change: term changed from 10 to 11, leader changed from 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66) to 5f827e4e0adc48ca87a385c561e3c660 (127.6.166.65). New cstate: current_term: 11 leader_uuid: "5f827e4e0adc48ca87a385c561e3c660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } health_report { overall_health: HEALTHY } } }
17:14:21.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.176503  8955 consensus_peers.cc:487] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 -> Peer 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735): Couldn't send request to peer 13d4754fe07445a99ba61cb33e1d3b2a. Status: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
17:14:21.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:21.323048  9201 raft_consensus.cc:1261] T e6a7a34859d84fae9805e99114ca43db P dd20240671db45b885a8a5ca2364ec72 [term 11 FOLLOWER]: Refusing update from remote peer 5f827e4e0adc48ca87a385c561e3c660: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
17:14:21.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:21.324451  9259 consensus_queue.cc:1035] T e6a7a34859d84fae9805e99114ca43db P 5f827e4e0adc48ca87a385c561e3c660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
17:14:21.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet e6a7a34859d84fae9805e99114ca43db for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837' with partition [<start>, <end>)
17:14:21.375 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e6a7a34859d84fae9805e99114ca43db@[13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735),5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)[L],dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)]}]
17:14:21.375 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e6a7a34859d84fae9805e99114ca43db, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1738), Traces: [0ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [2ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [2ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [18ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [19ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [21ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [21ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [38ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [40ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [41ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [41ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [78ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [80ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [81ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [81ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [195ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [199ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [200ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [202ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [202ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [215ms] refreshing cache from master, [215ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [218ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [219ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [221ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [221ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [1095ms] refreshing cache from master, [1095ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1100ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, [1101ms] sending RPC to server 13d4754fe07445a99ba61cb33e1d3b2a, [1103ms] received response from server 13d4754fe07445a99ba61cb33e1d3b2a: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [1103ms] delaying RPC due to: Network error: Failed to connect to peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735): Connection refused: /127.6.166.66:35735, [1735ms] refreshing cache from master, [1735ms] Sub RPC GetTableLocations: sending RPC to server master-127.6.166.124:44315, [1738ms] Sub RPC GetTableLocations: received response from server master-127.6.166.124:44315: OK, deferred=Deferred@1922799331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/597345472@20b78537 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305) -> (continuation of Deferred@1899505494 after retry RPC@662905036) -> (continuation of Deferred@1697088421 after retry RPC@1646068) -> (continuation of Deferred@963475008 after retry RPC@1909668388), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7cb07539 -> wakeup thread Time-limited test -> (continuation of Deferred@1666401162 after retry RPC@1144736046) -> (continuation of Deferred@2005211125 after retry RPC@1768509970) -> (continuation of Deferred@2048839555 after retry RPC@260783305) -> (continuation of Deferred@1899505494 after retry RPC@662905036) -> (continuation of Deferred@1697088421 after retry RPC@1646068) -> (continuation of Deferred@963475008 after retry RPC@1909668388))) after lookup
17:14:21.376 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] connecting to peer
17:14:21.377 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] Successfully connected to peer
17:14:21.396 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x2ae4dabd, L:/127.0.0.1:41312 - R:/127.6.166.65:45345] using TOKEN/null
17:14:21.435 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.6.166.66:35735
17:14:21.435 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:21.437041  6809 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:21.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:21.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
17:14:21.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.751129  9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
17:14:21.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.751780  9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
17:14:21.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.752449  9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
17:14:21.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.784677  9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
17:14:21.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.785604  9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.166.66
17:14:21.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:21.821445  9290 tablet_server_runner.cc:78] Tablet server non-default flags:
17:14:21.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.166.84:37267
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.166.66:35735
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.166.66
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40641
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.166.126:39943,127.6.166.125:39419,127.6.166.124:44315
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
17:14:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.166.66
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/logs
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 16:08:30 UTC on ccycloud.quasar-mtkpji.root.comops.site
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4493844
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
17:14:21.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:21.822944  9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
17:14:21.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:21.824972  9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
17:14:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.844229  9297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:22.264158  8955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.166.66:35735: connect: Connection refused (error 111) [suppressed 4 similar messages]
17:14:23.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.844434  9299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:23.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:21.846470  9290 server_base.cc:1034] running on GCE node
17:14:23.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 17:14:21.844683  9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
17:14:23.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.107932  9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
17:14:23.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.110410  9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
17:14:23.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.111799  9290 hybrid_clock.cc:648] HybridClock initialized: now 1783530863111778 us; error 57 us; skew 500 ppm
17:14:23.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.112646  9290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
17:14:23.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.119412  9290 webserver.cc:460] Webserver started at http://127.6.166.66:40641/ using document root <none> and password file <none>
17:14:23.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.120277  9290 fs_manager.cc:359] Metadata directory not provided
17:14:23.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.120472  9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
17:14:23.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.128006  9290 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.001s
17:14:23.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.132252  9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
17:14:23.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.133457  9290 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.133718  9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data,/tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "13d4754fe07445a99ba61cb33e1d3b2a"
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 17:13:31 on dist-test-slave-v72b"
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.135980  9290 fs_report.cc:383] FS layout report
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/wal
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/data
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
17:14:23.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
17:14:23.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.177522  9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
17:14:23.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.178925  9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
17:14:23.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.179391  9290 kserver.cc:163] Server-wide thread pool size limit: 3276
17:14:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.181807  9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
17:14:23.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.188438  9317 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
17:14:23.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.196065  9290 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
17:14:23.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.196285  9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s	user 0.002s	sys 0.000s
17:14:23.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.196507  9290 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
17:14:23.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.200953  9290 ts_tablet_manager.cc:610] Registered 1 tablets
17:14:23.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.201128  9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
17:14:23.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.201598  9317 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap starting.
17:14:23.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.293977  9317 log.cc:824] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Log is configured to *not* fsync() on all Append() calls
17:14:23.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.413681  9290 rpc_server.cc:307] RPC server started. Bound to: 127.6.166.66:35735
17:14:23.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.413920  9424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.166.66:35735 every 8 connection(s)
17:14:23.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.416550  9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMwcjhq/test-tmp/mini-kudu-cluster2862207107120246509/ts-1/data/info.pb
17:14:23.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.418895  6809 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu as pid 9290
17:14:23.419 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
17:14:23.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.426298  9317 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
17:14:23.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.427786  9317 tablet_bootstrap.cc:492] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Bootstrap complete.
17:14:23.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.430320  9317 ts_tablet_manager.cc:1397] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent bootstrapping tablet: real 0.229s	user 0.189s	sys 0.027s
17:14:23.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.464776  9317 raft_consensus.cc:348] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:23.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.469755  9317 raft_consensus.cc:729] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d4754fe07445a99ba61cb33e1d3b2a, State: Initialized, Role: FOLLOWER
17:14:23.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.471329  9317 consensus_queue.cc:260] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d4754fe07445a99ba61cb33e1d3b2a" member_type: VOTER last_known_addr { host: "127.6.166.66" port: 35735 } } peers { permanent_uuid: "dd20240671db45b885a8a5ca2364ec72" member_type: VOTER last_known_addr { host: "127.6.166.67" port: 35283 } } peers { permanent_uuid: "5f827e4e0adc48ca87a385c561e3c660" member_type: VOTER last_known_addr { host: "127.6.166.65" port: 45345 } }
17:14:23.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.482689  9317 ts_tablet_manager.cc:1428] T e6a7a34859d84fae9805e99114ca43db P 13d4754fe07445a99ba61cb33e1d3b2a: Time spent starting tablet: real 0.052s	user 0.034s	sys 0.012s
17:14:23.496 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:14:23.520 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783530813837, tablet=e6a7a34859d84fae9805e99114ca43db, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
17:14:23.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.527729  9425 heartbeater.cc:347] Connected to a master server at 127.6.166.124:44315
17:14:23.528 [DEBUG - main] (Connection.java:575) [peer master-127.6.166.124:44315(127.6.166.124:44315)] disconnecting while in state READY
17:14:23.529 [DEBUG - main] (Connection.java:575) [peer dd20240671db45b885a8a5ca2364ec72(127.6.166.67:35283)] disconnecting while in state TERMINATED
17:14:23.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.528258  9425 heartbeater.cc:464] Registering TS with master...
17:14:23.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.529501  9425 heartbeater.cc:511] Master 127.6.166.124:44315 requested a full tablet report, sending...
17:14:23.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.530337  9427 heartbeater.cc:347] Connected to a master server at 127.6.166.126:39943
17:14:23.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.530668  9427 heartbeater.cc:464] Registering TS with master...
17:14:23.532 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.6.166.124:44315(127.6.166.124:44315)] handling channelInactive
17:14:23.533 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.6.166.124:44315(127.6.166.124:44315)] cleaning up while in state READY due to: connection closed
17:14:23.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.534047  7007 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:14:23.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.537151  7007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.166.66:52083
17:14:23.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.537693  9426 heartbeater.cc:347] Connected to a master server at 127.6.166.125:39419
17:14:23.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.538018  9426 heartbeater.cc:464] Registering TS with master...
17:14:23.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.536715  9427 heartbeater.cc:511] Master 127.6.166.126:39943 requested a full tablet report, sending...
17:14:23.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.538873  9426 heartbeater.cc:511] Master 127.6.166.125:39419 requested a full tablet report, sending...
17:14:23.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.543404  6929 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:14:23.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.543984  6854 ts_manager.cc:194] Re-registered known tserver with Master: 13d4754fe07445a99ba61cb33e1d3b2a (127.6.166.66:35735)
17:14:23.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.544030  9425 heartbeater.cc:503] Master 127.6.166.124:44315 was elected leader, sending a full tablet report...
17:14:23.550 [DEBUG - main] (Connection.java:575) [peer master-127.6.166.126:39943(127.6.166.126:39943)] disconnecting while in state READY
17:14:23.550 [DEBUG - main] (Connection.java:575) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] disconnecting while in state READY
17:14:23.551 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.6.166.126:39943(127.6.166.126:39943)] handling channelInactive
17:14:23.551 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.6.166.126:39943(127.6.166.126:39943)] cleaning up while in state READY due to: connection closed
17:14:23.552 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] handling channelInactive
17:14:23.553 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5f827e4e0adc48ca87a385c561e3c660(127.6.166.65:45345)] cleaning up while in state READY due to: connection closed
17:14:23.553 [DEBUG - main] (Connection.java:575) [peer 13d4754fe07445a99ba61cb33e1d3b2a(127.6.166.66:35735)] disconnecting while in state TERMINATED
17:14:23.553 [DEBUG - main] (Connection.java:575) [peer master-127.6.166.125:39419(127.6.166.125:39419)] disconnecting while in state READY
17:14:23.555 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
17:14:23.557 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.6.166.125:39419(127.6.166.125:39419)] handling channelInactive
17:14:23.557 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.6.166.125:39419(127.6.166.125:39419)] cleaning up while in state READY due to: connection closed
17:14:23.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.559194  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 8930
17:14:23.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.590250  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 9290
17:14:23.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.615321  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 9110
17:14:23.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.644086  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 6821
17:14:23.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.676644  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 6891
17:14:23.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 17:14:23.707010  6809 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMwcjhq/build/tsan/bin/kudu with pid 6966
17:14:24.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T17:14:24Z chronyd exiting

Time: 66.798

OK (1 test)

