JUnit version 4.13.2
.19:44:05.653 [INFO - main] (RandomUtils.java:49) Using random seed: 1786477445647
19:44:05.706 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
19:44:05.770 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
19:44:05.787 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskE52uFS/test-tmp
19:44:05.790 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
19:44:05.795 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskE52uFS/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
19:44:06.089 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

19:44:06.266 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
19:44:06.268 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

19:44:06.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.274629  5031 mini_oidc.cc:152] Starting JWKS server
19:44:06.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.279685  5031 webserver.cc:460] Webserver started at http://127.0.0.1:43653/ using document root <none> and password file <none>
19:44:06.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.280285  5031 mini_oidc.cc:189] Starting OIDC Discovery server
19:44:06.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.282582  5031 webserver.cc:460] Webserver started at http://127.0.0.1:44395/ using document root <none> and password file <none>
19:44:06.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T19:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
19:44:06.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T19:44:06Z Disabled control of system clock
19:44:06.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.347654  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:06.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:06.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/wal
19:44:06.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data
19:44:06.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:06.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:06.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:06.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:06.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:06.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:06.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/logs
19:44:06.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data/info.pb
19:44:06.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:06.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:06.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:06.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:06.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:06.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:06.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:06.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:06.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:44:06.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:06.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:44:06.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:44:06.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.254:36095
19:44:06.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.254
19:44:06.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:06.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:06.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:06.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:06.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:06.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:44:06.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:06.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:44:06.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43653/jwks.json with env {}
19:44:06.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.714233  5055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:06.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.714957  5055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:06.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.715577  5055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:06.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.748494  5055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:44:06.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.748843  5055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:06.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.749051  5055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:44:06.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.749336  5055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:44:06.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.749547  5055 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:44:06.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.749758  5055 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:43653/jwks.json
19:44:06.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.795706  5055 master_runner.cc:386] Master server non-default flags:
19:44:06.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:06.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:06.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:06.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:06.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data
19:44:06.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/wal
19:44:06.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:44:06.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.254:36095
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43653/jwks.json
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data/info.pb
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.254
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:06.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/logs
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:06.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:06.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:06.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:06.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:06.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:06.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.797614  5055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:06.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.799719  5055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:06.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.822021  5065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:08.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.822135  5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:06.822208  5064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:06.824559  5055 server_base.cc:1034] running on GCE node
19:44:08.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.058984  5055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:08.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.062441  5055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:08.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.063844  5055 hybrid_clock.cc:648] HybridClock initialized: now 1786477448063773 us; error 98 us; skew 500 ppm
19:44:08.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.064810  5055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:08.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.077069  5055 webserver.cc:460] Webserver started at http://127.4.233.254:39403/ using document root <none> and password file <none>
19:44:08.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.078344  5055 fs_manager.cc:359] Metadata directory not provided
19:44:08.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.078560  5055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:08.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.079026  5055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:44:08.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.085112  5055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data/instance:
19:44:08.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d5162eca18746abaf96b1c9f28749ad"
19:44:08.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:08 on dist-test-slave-z8x9"
19:44:08.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.086668  5055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/wal/instance:
19:44:08.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d5162eca18746abaf96b1c9f28749ad"
19:44:08.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:08 on dist-test-slave-z8x9"
19:44:08.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.095674  5055 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.006s	sys 0.003s
19:44:08.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.103428  5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:08.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.104905  5055 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.002s
19:44:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.105389  5055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/wal
19:44:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3d5162eca18746abaf96b1c9f28749ad"
19:44:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:08 on dist-test-slave-z8x9"
19:44:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.105948  5055 fs_report.cc:383] FS layout report
19:44:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/wal
19:44:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/wal
19:44:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data/data
19:44:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:08.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:08.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:08.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:08.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.154553  5055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.156714  5055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.157430  5055 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:08.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.254570  5055 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.254:36095
19:44:08.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.254654  5174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.254:36095 every 8 connection(s)
19:44:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.258108  5055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/data/info.pb
19:44:08.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.264128  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 5055
19:44:08.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.264794  5031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-0/wal/instance
19:44:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.265796  5176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.270866  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/wal
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/logs
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data/info.pb
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:08.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.253:41097
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.253
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:44:08.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43653/jwks.json with env {}
19:44:08.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.285789  5176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
19:44:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.325146  5176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } has no permanent_uuid. Determining permanent_uuid...
19:44:08.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.328117  5087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:08.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.333643  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.253:41097: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.383437  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } attempt: 1
19:44:08.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.388652  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.253:41097: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:08.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.457278  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } attempt: 2
19:44:08.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.461957  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.253:41097: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:08.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.553566  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } attempt: 3
19:44:08.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.558735  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.253:41097: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:08.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.666195  5180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:08.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.667279  5180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:08.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.668255  5180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:08.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.702239  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } attempt: 4
19:44:08.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.704562  5180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:44:08.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.705029  5180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:08.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.705399  5180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:44:08.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.705662  5180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:44:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.706027  5180 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:44:08.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.706308  5180 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:43653/jwks.json
19:44:08.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.707304  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.253:41097: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.746431  5180 master_runner.cc:386] Master server non-default flags:
19:44:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/wal
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.253:41097
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43653/jwks.json
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data/info.pb
19:44:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.253
19:44:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/logs
19:44:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:44:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:08.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:08.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:08.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:08.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:08.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:08.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.748206  5180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:08.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.750226  5180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:08.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.775648  5199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:09.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:09.006872  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } attempt: 5
19:44:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:09.012998  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.253:41097: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:09.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:09.560611  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } attempt: 6
19:44:09.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:09.569149  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.253:41097: Network error: Client connection negotiation failed: client connection to 127.4.233.253:41097: connect: Connection refused (error 111)
19:44:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.775728  5203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:08.775712  5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:08.777721  5180 server_base.cc:1034] running on GCE node
19:44:10.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.032473  5180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:10.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.036765  5180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:10.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.038285  5180 hybrid_clock.cc:648] HybridClock initialized: now 1786477450038246 us; error 76 us; skew 500 ppm
19:44:10.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.039630  5180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:10.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.048849  5180 webserver.cc:460] Webserver started at http://127.4.233.253:36219/ using document root <none> and password file <none>
19:44:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.050275  5180 fs_manager.cc:359] Metadata directory not provided
19:44:10.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.050597  5180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:10.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.051280  5180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:44:10.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.057729  5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data/instance:
19:44:10.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdb95c3584ed44e493f3acd23ce0a683"
19:44:10.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:10 on dist-test-slave-z8x9"
19:44:10.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.059154  5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/wal/instance:
19:44:10.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdb95c3584ed44e493f3acd23ce0a683"
19:44:10.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:10 on dist-test-slave-z8x9"
19:44:10.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.067904  5180 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.009s	sys 0.000s
19:44:10.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.074538  5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:10.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.075726  5180 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
19:44:10.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.076128  5180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/wal
19:44:10.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdb95c3584ed44e493f3acd23ce0a683"
19:44:10.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:10 on dist-test-slave-z8x9"
19:44:10.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.076550  5180 fs_report.cc:383] FS layout report
19:44:10.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:10.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/wal
19:44:10.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/wal
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data/data
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:10.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:10.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:10.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:10.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:10.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:10.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.125787  5180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:10.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.127928  5180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:10.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.128575  5180 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:10.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.224663  5180 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.253:41097
19:44:10.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.224788  5313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.253:41097 every 8 connection(s)
19:44:10.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.228395  5180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/data/info.pb
19:44:10.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.229795  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 5180
19:44:10.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.230314  5031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-1/wal/instance
19:44:10.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.233385  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:10.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:10.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/wal
19:44:10.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/logs
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data/info.pb
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.252:42637
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.252
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:10.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:10.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:10.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:44:10.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:10.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:44:10.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43653/jwks.json with env {}
19:44:10.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.237262  5317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:44:10.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.258281  5317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
19:44:10.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.297971  5317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } has no permanent_uuid. Determining permanent_uuid...
19:44:10.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.320432  5317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
19:44:10.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.323309  5232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.328425  5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.378026  5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 1
19:44:10.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.382748  5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.451181  5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 2
19:44:10.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.457419  5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.549022  5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 3
19:44:10.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.553606  5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.629702  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } attempt: 7
19:44:10.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.644600  5176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
19:44:10.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.650945  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.657642  5319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:10.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.658304  5319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:10.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.658808  5319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:10.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.696897  5319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:44:10.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.697395  5319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:10.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.697079  5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 4
19:44:10.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.697755  5319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:44:10.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.698118  5319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:44:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.698570  5319 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:44:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.698820  5319 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:43653/jwks.json
19:44:10.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.701895  5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.709488  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 1
19:44:10.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.713764  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.739301  5319 master_runner.cc:386] Master server non-default flags:
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/wal
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.252:42637
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:44:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43653/jwks.json
19:44:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data/info.pb
19:44:10.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.252
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/logs
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:10.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.741005  5319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.743875  5319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:10.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.771894  5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:10.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.795305  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 2
19:44:10.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.801049  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:10.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.886698  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 3
19:44:10.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.892557  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:11.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:11.001386  5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 5
19:44:11.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:11.006762  5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:11.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:11.033077  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 4
19:44:11.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:11.039222  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:11.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:11.323136  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 5
19:44:11.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:11.328817  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:11.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:11.554292  5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 6
19:44:11.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:11.562994  5317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:11.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:11.881310  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 6
19:44:11.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:11.891371  5176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.233.252:42637: Network error: Client connection negotiation failed: client connection to 127.4.233.252:42637: connect: Connection refused (error 111)
19:44:12.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.773123  5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:12.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:10.773589  5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:12.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:10.774638  5319 server_base.cc:1034] running on GCE node
19:44:12.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.036468  5319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:12.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.039767  5319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:12.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.041143  5319 hybrid_clock.cc:648] HybridClock initialized: now 1786477452041112 us; error 67 us; skew 500 ppm
19:44:12.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.042376  5319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:12.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.051335  5319 webserver.cc:460] Webserver started at http://127.4.233.252:37537/ using document root <none> and password file <none>
19:44:12.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.052495  5319 fs_manager.cc:359] Metadata directory not provided
19:44:12.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.052711  5319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:12.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.053229  5319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:44:12.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.059113  5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data/instance:
19:44:12.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6381160c10214bed9e2a5f02196ccbf7"
19:44:12.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:12 on dist-test-slave-z8x9"
19:44:12.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.060372  5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/wal/instance:
19:44:12.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6381160c10214bed9e2a5f02196ccbf7"
19:44:12.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:12 on dist-test-slave-z8x9"
19:44:12.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.069701  5319 fs_manager.cc:691] Time spent creating directory manager: real 0.009s	user 0.010s	sys 0.001s
19:44:12.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.076514  5373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:12.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.078114  5319 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.001s
19:44:12.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.078572  5319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/wal
19:44:12.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6381160c10214bed9e2a5f02196ccbf7"
19:44:12.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:12 on dist-test-slave-z8x9"
19:44:12.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.079070  5319 fs_report.cc:383] FS layout report
19:44:12.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:12.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/wal
19:44:12.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/wal
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data/data
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:12.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:12.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.118970  5319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:12.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.120802  5319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:12.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.121379  5319 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:12.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.217499  5319 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.252:42637
19:44:12.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.217608  5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.252:42637 every 8 connection(s)
19:44:12.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.221020  5319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/data/info.pb
19:44:12.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.222101  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 5319
19:44:12.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.222633  5031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/master-2/wal/instance
19:44:12.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.230391  5467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:44:12.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.253055  5467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
19:44:12.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.293459  5467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } has no permanent_uuid. Determining permanent_uuid...
19:44:12.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.307726  5467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
19:44:12.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.327898  5467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7: Bootstrap starting.
19:44:12.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.336980  5467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7: Neither blocks nor log segments found. Creating new log.
19:44:12.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.339280  5467 log.cc:824] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7: Log is configured to *not* fsync() on all Append() calls
19:44:12.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.345885  5467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7: No bootstrap required, opened a new log
19:44:12.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.370329  5467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:12.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.371083  5467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:44:12.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.371366  5467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6381160c10214bed9e2a5f02196ccbf7, State: Initialized, Role: FOLLOWER
19:44:12.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.372419  5467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:12.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.375854  5495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:12.377633  5383 tablet.cc:2369] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:44:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.378160  5467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: configured and running, proceeding with master startup.
19:44:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.380607  5495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: This master's current role is: FOLLOWER
19:44:12.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:12.413995  5516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
19:44:12.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:12.415201  5516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
19:44:12.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.407953  5319 master_runner.cc:186] Error getting master registration for 127.4.233.254:36095: OK, instance_id { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" instance_seqno: 1786477448229018 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
19:44:12.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.438966  5319 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:44:12.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.523568  5495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:44:12.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.524160  5495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:12.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.527022  5495 leader_election.cc:288] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d5162eca18746abaf96b1c9f28749ad (127.4.233.254:36095), bdb95c3584ed44e493f3acd23ce0a683 (127.4.233.253:41097)
19:44:12.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.623696  5317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 7
19:44:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.650561  5317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: Bootstrap starting.
19:44:12.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.663175  5317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: Neither blocks nor log segments found. Creating new log.
19:44:12.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.666152  5317 log.cc:824] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: Log is configured to *not* fsync() on all Append() calls
19:44:12.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.675576  5317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: No bootstrap required, opened a new log
19:44:12.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.703464  5317 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:12.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.704432  5317 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:44:12.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.704736  5317 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb95c3584ed44e493f3acd23ce0a683, State: Initialized, Role: FOLLOWER
19:44:12.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.706058  5317 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:12.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.709519  5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:12.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.710520  5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [sys.catalog]: This master's current role is: FOLLOWER
19:44:12.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.711674  5317 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [sys.catalog]: configured and running, proceeding with master startup.
19:44:12.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.711997  5278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6381160c10214bed9e2a5f02196ccbf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb95c3584ed44e493f3acd23ce0a683" is_pre_election: true
19:44:12.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.713208  5278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6381160c10214bed9e2a5f02196ccbf7 in term 0.
19:44:12.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.715075  5376 leader_election.cc:302] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6381160c10214bed9e2a5f02196ccbf7, bdb95c3584ed44e493f3acd23ce0a683; no voters: 
19:44:12.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.716413  5495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
19:44:12.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.716878  5495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
19:44:12.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.717309  5495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 0 FOLLOWER]: Advancing to term 1
19:44:12.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.724865  5495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:12.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.727167  5495 leader_election.cc:288] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [CANDIDATE]: Term 1 election: Requested vote from peers 3d5162eca18746abaf96b1c9f28749ad (127.4.233.254:36095), bdb95c3584ed44e493f3acd23ce0a683 (127.4.233.253:41097)
19:44:12.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.731850  5278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6381160c10214bed9e2a5f02196ccbf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb95c3584ed44e493f3acd23ce0a683"
19:44:12.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.732613  5278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [term 0 FOLLOWER]: Advancing to term 1
19:44:12.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.740952  5278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6381160c10214bed9e2a5f02196ccbf7 in term 1.
19:44:12.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.743148  5376 leader_election.cc:302] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6381160c10214bed9e2a5f02196ccbf7, bdb95c3584ed44e493f3acd23ce0a683; no voters: 
19:44:12.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.745157  5495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 1 FOLLOWER]: Leader election won for term 1
19:44:12.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.748373  5495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [term 1 LEADER]: Becoming Leader. State: Replica: 6381160c10214bed9e2a5f02196ccbf7, State: Running, Role: LEADER
19:44:12.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.750597  5495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:12.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:12.777899  5544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
19:44:12.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:12.785902  5544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
19:44:12.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.805474  5532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6381160c10214bed9e2a5f02196ccbf7. Latest consensus state: current_term: 1 leader_uuid: "6381160c10214bed9e2a5f02196ccbf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:12.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.809674  5532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: This master's current role is: LEADER
19:44:12.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.826678  5547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
19:44:12.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.771294  5180 master_runner.cc:186] Error getting master registration for 127.4.233.254:36095: OK, instance_id { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" instance_seqno: 1786477448229018 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
19:44:12.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.831496  5180 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:44:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.835366  5547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
19:44:12.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.863570  5278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [term 1 FOLLOWER]: Refusing update from remote peer 6381160c10214bed9e2a5f02196ccbf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:44:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.865770  5532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
19:44:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:12.888487  5237 tablet.cc:2369] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:44:12.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.911208  5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6381160c10214bed9e2a5f02196ccbf7. Latest consensus state: current_term: 1 leader_uuid: "6381160c10214bed9e2a5f02196ccbf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:12.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.912228  5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [sys.catalog]: This master's current role is: FOLLOWER
19:44:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.923532  5548 mvcc.cc:205] Tried to move back new op lower bound from 7317411646909206528 to 7317411646481416192. Current Snapshot: MvccSnapshot[applied={T|T < 7317411646909206528}]
19:44:12.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.924826  5176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } attempt: 7
19:44:12.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.926967  5495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6381160c10214bed9e2a5f02196ccbf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:12.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.927776  5495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: This master's current role is: LEADER
19:44:12.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.957433  5176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad: Bootstrap starting.
19:44:12.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.970244  5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6381160c10214bed9e2a5f02196ccbf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:12.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.971063  5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683 [sys.catalog]: This master's current role is: FOLLOWER
19:44:12.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.977267  5176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad: Neither blocks nor log segments found. Creating new log.
19:44:12.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.977286  5547 catalog_manager.cc:1347] Generated new cluster ID: e2a4d66b9f584055aa863e2aba45442b
19:44:12.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.977677  5547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
19:44:12.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.980908  5176 log.cc:824] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad: Log is configured to *not* fsync() on all Append() calls
19:44:12.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:12.995797  5176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad: No bootstrap required, opened a new log
19:44:13.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.032995  5176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:13.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.034520  5176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:44:13.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.035065  5176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5162eca18746abaf96b1c9f28749ad, State: Initialized, Role: FOLLOWER
19:44:13.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.036490  5176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } }
19:44:13.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.056355  5176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [sys.catalog]: configured and running, proceeding with master startup.
19:44:13.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.056655  5128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6381160c10214bed9e2a5f02196ccbf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5162eca18746abaf96b1c9f28749ad" is_pre_election: true
19:44:13.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.056598  5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6381160c10214bed9e2a5f02196ccbf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5162eca18746abaf96b1c9f28749ad"
19:44:13.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.058144  5125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [term 0 FOLLOWER]: Advancing to term 1
19:44:13.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.057765  5565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:13.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.063022  5565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [sys.catalog]: This master's current role is: FOLLOWER
19:44:13.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.069484  5125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [term 1 FOLLOWER]: Refusing update from remote peer 6381160c10214bed9e2a5f02196ccbf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:44:13.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.083446  5532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
19:44:13.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.175697  5547 catalog_manager.cc:1370] Generated new certificate authority record
19:44:13.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.180464  5547 catalog_manager.cc:1504] Loading token signing keys...
19:44:13.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.218743  5565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6381160c10214bed9e2a5f02196ccbf7. Latest consensus state: current_term: 1 leader_uuid: "6381160c10214bed9e2a5f02196ccbf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:13.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.219658  5565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [sys.catalog]: This master's current role is: FOLLOWER
19:44:13.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.224049  5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6381160c10214bed9e2a5f02196ccbf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:13.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.225055  5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad [sys.catalog]: This master's current role is: FOLLOWER
19:44:13.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.234069  5495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6381160c10214bed9e2a5f02196ccbf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d5162eca18746abaf96b1c9f28749ad" member_type: VOTER last_known_addr { host: "127.4.233.254" port: 36095 } } peers { permanent_uuid: "bdb95c3584ed44e493f3acd23ce0a683" member_type: VOTER last_known_addr { host: "127.4.233.253" port: 41097 } } peers { permanent_uuid: "6381160c10214bed9e2a5f02196ccbf7" member_type: VOTER last_known_addr { host: "127.4.233.252" port: 42637 } } }
19:44:13.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.234899  5495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7 [sys.catalog]: This master's current role is: LEADER
19:44:13.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.293709  5547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6381160c10214bed9e2a5f02196ccbf7: Generated new TSK 0
19:44:13.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.295270  5547 catalog_manager.cc:1514] Initializing in-progress tserver states...
19:44:13.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.318295  5590 catalog_manager.cc:1259] Loaded cluster ID: e2a4d66b9f584055aa863e2aba45442b
19:44:13.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.318656  5590 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad: loading cluster ID for follower catalog manager: success
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.346841  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:13.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:0
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:13.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:13.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.355049  5590 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad: acquiring CA information for follower catalog manager: success
19:44:13.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.384073  5590 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d5162eca18746abaf96b1c9f28749ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:44:13.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.794219  5544 catalog_manager.cc:1259] Loaded cluster ID: e2a4d66b9f584055aa863e2aba45442b
19:44:13.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.795056  5544 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: loading cluster ID for follower catalog manager: success
19:44:13.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.823865  5544 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: acquiring CA information for follower catalog manager: success
19:44:13.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:13.842948  5544 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bdb95c3584ed44e493f3acd23ce0a683: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:44:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:13.934317  5593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:13.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:13.935876  5593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:13.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:13.936998  5593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:13.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:13.998515  5593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:14.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:14.000603  5593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.193
19:44:14.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:14.068135  5593 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:0
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:14.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:14.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:14.072199  5593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:14.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:14.075290  5593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:14.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:14.133209  5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:16.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:14.145893  5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:16.667434  5593 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 2.543s	user 0.602s	sys 0.944s
19:44:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:16.700896  5593 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 2.577s	user 0.602s	sys 0.945s
19:44:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.701220  5593 server_base.cc:1034] running on GCE node
19:44:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:15.509972  5602 debug-util.cc:398] Leaking SignalData structure 0x7b0800014860 after lost signal to thread 5593
19:44:16.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:16.703352  5607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:16.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.705299  5593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:16.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.708760  5593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:16.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.710305  5593 hybrid_clock.cc:648] HybridClock initialized: now 1786477456710089 us; error 227 us; skew 500 ppm
19:44:16.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.711584  5593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:16.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.728799  5593 webserver.cc:460] Webserver started at http://127.4.233.193:44387/ using document root <none> and password file <none>
19:44:16.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.730357  5593 fs_manager.cc:359] Metadata directory not provided
19:44:16.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.730715  5593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:16.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.731390  5593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:44:16.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.737526  5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/instance:
19:44:16.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:16.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:16 on dist-test-slave-z8x9"
19:44:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.739002  5593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal/instance:
19:44:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:16 on dist-test-slave-z8x9"
19:44:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.748903  5593 fs_manager.cc:691] Time spent creating directory manager: real 0.009s	user 0.010s	sys 0.000s
19:44:16.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.756723  5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:16.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.758453  5593 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.007s	sys 0.000s
19:44:16.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.758921  5593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:16.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:16.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:16 on dist-test-slave-z8x9"
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.759387  5593 fs_report.cc:383] FS layout report
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/data
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:16.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:16.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.822544  5593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:16.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.824342  5593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:16.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.824867  5593 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:16.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.828032  5593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:16.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.833593  5593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
19:44:16.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.833853  5593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
19:44:16.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.834200  5593 ts_tablet_manager.cc:610] Registered 0 tablets
19:44:16.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:16.834395  5593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
19:44:17.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.020924  5593 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.193:33549
19:44:17.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.021044  5847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.193:33549 every 8 connection(s)
19:44:17.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.024142  5593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:17.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.034570  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 5593
19:44:17.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.035462  5031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal/instance
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.061746  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:17.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:0
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:17.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:17.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.084928  5857 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:17.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.085579  5857 heartbeater.cc:464] Registering TS with master...
19:44:17.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.086547  5859 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:17.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.086978  5859 heartbeater.cc:464] Registering TS with master...
19:44:17.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.087414  5857 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:17.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.088331  5859 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:17.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.092696  5112 ts_manager.cc:194] Registered new tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:17.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.093698  5399 ts_manager.cc:194] Registered new tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:17.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.096588  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.193:33303
19:44:17.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.100961  5858 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:17.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.101420  5858 heartbeater.cc:464] Registering TS with master...
19:44:17.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.103142  5858 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:17.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.106755  5258 ts_manager.cc:194] Registered new tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:17.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.493417  5873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:17.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.494035  5873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:17.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.494776  5873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:17.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.530937  5873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.531867  5873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.194
19:44:17.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.570194  5873 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:17.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:0
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:17.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:17.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.571990  5873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:17.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:17.574855  5873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:17.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.600236  5888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:18.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.104506  5857 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:18.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:18.184727  5169 debug-util.cc:398] Leaking SignalData structure 0x7b0800072b80 after lost signal to thread 5056
19:44:18.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:18.186038  5169 debug-util.cc:398] Leaking SignalData structure 0x7b08000968a0 after lost signal to thread 5174
19:44:18.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.600270  5889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:18.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:17.601881  5893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:18.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:18.921641  5890 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:44:18.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.921706  5873 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:44:18.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.926579  5873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:18.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.930197  5873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:18.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.931959  5873 hybrid_clock.cc:648] HybridClock initialized: now 1786477458931911 us; error 86 us; skew 500 ppm
19:44:18.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.933015  5873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:18.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.941927  5873 webserver.cc:460] Webserver started at http://127.4.233.194:44553/ using document root <none> and password file <none>
19:44:18.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.943145  5873 fs_manager.cc:359] Metadata directory not provided
19:44:18.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.943432  5873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:18.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.944070  5873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:44:18.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.950418  5873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/instance:
19:44:18.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:18.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:18 on dist-test-slave-z8x9"
19:44:18.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.952066  5873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal/instance:
19:44:18.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:18.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:18 on dist-test-slave-z8x9"
19:44:18.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.962262  5873 fs_manager.cc:691] Time spent creating directory manager: real 0.009s	user 0.006s	sys 0.005s
19:44:18.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.970031  5903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:18.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.971501  5873 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.003s
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.971869  5873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:18 on dist-test-slave-z8x9"
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:18.972249  5873 fs_report.cc:383] FS layout report
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/data
19:44:18.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:19.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.042696  5873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.044639  5873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.045159  5873 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:19.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.049193  5873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:19.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.055473  5873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
19:44:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.055740  5873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
19:44:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.055958  5873 ts_tablet_manager.cc:610] Registered 0 tablets
19:44:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.056080  5873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
19:44:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.407172  5873 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.194:40309
19:44:19.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.407342  6132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.194:40309 every 8 connection(s)
19:44:19.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.411044  5873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.414288  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 5873
19:44:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.414919  5031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal/instance
19:44:19.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.443156  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:19.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:0
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:19.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.471163  6135 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:19.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.471858  6135 heartbeater.cc:464] Registering TS with master...
19:44:19.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.473583  6135 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:19.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.477471  5399 ts_manager.cc:194] Registered new tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:19.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.479902  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.194:38839
19:44:19.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.501130  6137 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:19.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.502411  6137 heartbeater.cc:464] Registering TS with master...
19:44:19.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.503310  6137 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:19.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.507105  5258 ts_manager.cc:194] Registered new tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:19.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.512069  6143 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:19.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.512594  6143 heartbeater.cc:464] Registering TS with master...
19:44:19.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.513877  6143 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:19.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:19.517894  5112 ts_manager.cc:194] Registered new tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:19.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:19.953056  6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:19.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:19.953809  6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:19.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:19.954312  6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:19.990015  6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:19.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:19.991117  6148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.195
19:44:20.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:20.034355  6148 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:0
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:44:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:20.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:20.036458  6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:20.038929  6148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:20.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:20.068563  6164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:20.487920  6135 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:21.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:20.068681  6167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:21.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:20.075659  6163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:21.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:21.537881  6165 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1467 milliseconds
19:44:21.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.538004  6148 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:44:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.539798  6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:21.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.542925  6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:21.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.544380  6148 hybrid_clock.cc:648] HybridClock initialized: now 1786477461544342 us; error 42 us; skew 500 ppm
19:44:21.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.545454  6148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:21.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.552780  6148 webserver.cc:460] Webserver started at http://127.4.233.195:45763/ using document root <none> and password file <none>
19:44:21.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.554000  6148 fs_manager.cc:359] Metadata directory not provided
19:44:21.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.554270  6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:21.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.554899  6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:44:21.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.560736  6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/instance:
19:44:21.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:21.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:21 on dist-test-slave-z8x9"
19:44:21.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.562152  6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal/instance:
19:44:21.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:21.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:21 on dist-test-slave-z8x9"
19:44:21.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.570765  6148 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.001s
19:44:21.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.577535  6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:21.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.578799  6148 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
19:44:21.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.579228  6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:21.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:21.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:21 on dist-test-slave-z8x9"
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.579631  6148 fs_report.cc:383] FS layout report
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/data
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:21.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:21.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.616357  6148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:21.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.618449  6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:21.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.619065  6148 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:21.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.622257  6148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:21.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.627270  6148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
19:44:21.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.627517  6148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
19:44:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.627899  6148 ts_tablet_manager.cc:610] Registered 0 tablets
19:44:21.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.628090  6148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
19:44:21.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.839972  6148 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.195:34997
19:44:21.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.840063  6388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.195:34997 every 8 connection(s)
19:44:21.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.842985  6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:21.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.851106  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 6148
19:44:21.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.851735  5031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal/instance
19:44:21.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.905521  6390 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:21.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.906191  6390 heartbeater.cc:464] Registering TS with master...
19:44:21.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.907748  6390 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:21.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.911157  5399 ts_manager.cc:194] Registered new tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:21.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.913934  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.195:59829
19:44:21.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.917593  6397 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:21.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.917973  6397 heartbeater.cc:464] Registering TS with master...
19:44:21.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.919647  6397 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.922748  5258 ts_manager.cc:194] Registered new tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:21.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.924260  6398 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:21.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.924561  6398 heartbeater.cc:464] Registering TS with master...
19:44:21.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.925482  6398 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:21.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.928198  5112 ts_manager.cc:194] Registered new tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:21.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:21.940042  5031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
19:44:21.953 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
19:44:21.955 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

19:44:21.967 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.4.233.254"
      port: 36095
    }
    bound_http_address {
      host: "127.4.233.254"
      port: 39403
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.4.233.253"
      port: 41097
    }
    bound_http_address {
      host: "127.4.233.253"
      port: 36219
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.4.233.252"
      port: 42637
    }
    bound_http_address {
      host: "127.4.233.252"
      port: 37537
    }
  }
}

19:44:21.973 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

19:44:21.977 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.4.233.193"
      port: 33549
    }
    bound_http_address {
      host: "127.4.233.193"
      port: 44387
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.4.233.194"
      port: 40309
    }
    bound_http_address {
      host: "127.4.233.194"
      port: 44553
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.4.233.195"
      port: 34997
    }
    bound_http_address {
      host: "127.4.233.195"
      port: 45763
    }
  }
}

19:44:21.977 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
19:44:22.189 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
19:44:22.453 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.4.233.253:41097(127.4.233.253:41097)] connecting to peer
19:44:22.538 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.4.233.252:42637(127.4.233.252:42637)] connecting to peer
19:44:22.540 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.4.233.254:36095(127.4.233.254:36095)] connecting to peer
19:44:22.578 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.4.233.253:41097(127.4.233.253:41097)] Successfully connected to peer
19:44:22.586 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.4.233.252:42637(127.4.233.252:42637)] Successfully connected to peer
19:44:22.675 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.4.233.254:36095(127.4.233.254:36095)] Successfully connected to peer
19:44:22.710 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.4.233.252
19:44:22.710 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.4.233.253
19:44:22.846 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.4.233.254
19:44:22.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:22.921324  6390 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:22.995 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x4f68db09, L:/127.0.0.1:40318 - R:/127.4.233.252:42637] using SASL/PLAIN
19:44:22.995 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x4f12ab3d, L:/127.0.0.1:36040 - R:/127.4.233.254:36095] using SASL/PLAIN
19:44:23.001 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x6b3bab01, L:/127.0.0.1:40194 - R:/127.4.233.253:41097] using SASL/PLAIN
19:44:23.054 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
19:44:23.055 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=667), Traces: [0ms] refreshing cache from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.233.253:41097, [144ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.233.252:42637, [145ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.233.254:36095, [621ms] Sub RPC ConnectToMaster: received response from server master-127.4.233.252:42637: OK, [621ms] Sub RPC ConnectToMaster: received response from server master-127.4.233.254:36095: OK, [624ms] Sub RPC ConnectToMaster: received response from server master-127.4.233.253:41097: OK, deferred=null) after lookup
19:44:23.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.123481  5399 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40318:
19:44:23.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373"
19:44:23.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
19:44:23.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
19:44:23.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
19:44:23.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
19:44:23.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
19:44:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
19:44:23.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
19:44:23.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
19:44:23.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
19:44:23.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
19:44:23.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:23.129879  5399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
19:44:23.224 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 4113d4b6754643fe85f46954d45a80d7
19:44:23.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.283551  6008 tablet_service.cc:1467] Processing CreateTablet for tablet c09d070664cb49608a8b69571121d26c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373 [id=4113d4b6754643fe85f46954d45a80d7]), partition=RANGE (key) PARTITION UNBOUNDED
19:44:23.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.285945  6008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09d070664cb49608a8b69571121d26c. 1 dirs total, 0 dirs full, 0 dirs failed
19:44:23.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.291214  6272 tablet_service.cc:1467] Processing CreateTablet for tablet c09d070664cb49608a8b69571121d26c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373 [id=4113d4b6754643fe85f46954d45a80d7]), partition=RANGE (key) PARTITION UNBOUNDED
19:44:23.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.293788  6272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09d070664cb49608a8b69571121d26c. 1 dirs total, 0 dirs full, 0 dirs failed
19:44:23.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.305892  5723 tablet_service.cc:1467] Processing CreateTablet for tablet c09d070664cb49608a8b69571121d26c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373 [id=4113d4b6754643fe85f46954d45a80d7]), partition=RANGE (key) PARTITION UNBOUNDED
19:44:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.308813  5723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09d070664cb49608a8b69571121d26c. 1 dirs total, 0 dirs full, 0 dirs failed
19:44:23.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.340473  6457 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap starting.
19:44:23.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.351847  6457 tablet_bootstrap.cc:654] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Neither blocks nor log segments found. Creating new log.
19:44:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.355090  6457 log.cc:824] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Log is configured to *not* fsync() on all Append() calls
19:44:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.355393  6458 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap starting.
19:44:23.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.362584  6462 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap starting.
19:44:23.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.363495  6457 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: No bootstrap required, opened a new log
19:44:23.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.364140  6457 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent bootstrapping tablet: real 0.024s	user 0.004s	sys 0.014s
19:44:23.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.365739  6458 tablet_bootstrap.cc:654] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Neither blocks nor log segments found. Creating new log.
19:44:23.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.368942  6458 log.cc:824] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Log is configured to *not* fsync() on all Append() calls
19:44:23.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.370303  6462 tablet_bootstrap.cc:654] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Neither blocks nor log segments found. Creating new log.
19:44:23.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.373715  6462 log.cc:824] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Log is configured to *not* fsync() on all Append() calls
19:44:23.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.376477  6458 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: No bootstrap required, opened a new log
19:44:23.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.377223  6458 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.006s
19:44:23.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.381376  6462 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: No bootstrap required, opened a new log
19:44:23.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.382107  6462 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.012s
19:44:23.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.402199  6457 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.403378  6457 raft_consensus.cc:374] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:44:23.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.403710  6457 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Initialized, Role: FOLLOWER
19:44:23.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.404976  6457 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.417091  6462 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.418272  6462 raft_consensus.cc:374] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:44:23.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.418941  6462 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd83ec522d79445cb7c23e85a2486c50, State: Initialized, Role: FOLLOWER
19:44:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.420248  6462 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.422098  6457 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent starting tablet: real 0.058s	user 0.029s	sys 0.030s
19:44:23.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.421785  6458 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.422958  6458 raft_consensus.cc:374] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:44:23.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.423281  6458 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Initialized, Role: FOLLOWER
19:44:23.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.424456  6458 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.435897  6458 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent starting tablet: real 0.058s	user 0.046s	sys 0.003s
19:44:23.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.462502  6462 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent starting tablet: real 0.080s	user 0.037s	sys 0.021s
19:44:23.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:23.508859  6146 tablet.cc:2369] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:44:23.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:23.550603  5863 tablet.cc:2369] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:44:23.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.596063  6468 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:44:23.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.596714  6468 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.600544  6468 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:23.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.618212  6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50" is_pre_election: true
19:44:23.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.619560  6296 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 0.
19:44:23.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.621277  5911 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50; no voters: 
19:44:23.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.621935  5760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a" is_pre_election: true
19:44:23.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.622403  6468 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Leader pre-election won for term 1
19:44:23.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.622774  6468 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
19:44:23.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.622942  5760 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 0.
19:44:23.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.623147  6468 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 0 FOLLOWER]: Advancing to term 1
19:44:23.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.628422  6468 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.630223  6468 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 1 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:23.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.631462  6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:23.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.631536  5760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:23.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.632123  5760 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 0 FOLLOWER]: Advancing to term 1
19:44:23.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.632157  6296 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 0 FOLLOWER]: Advancing to term 1
19:44:23.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.637727  6296 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 1.
19:44:23.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.637707  5760 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 1.
19:44:23.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.638840  5911 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a; no voters: 
19:44:23.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.639765  6468 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Leader election won for term 1
19:44:23.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.642202  6468 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 LEADER]: Becoming Leader. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Running, Role: LEADER
19:44:23.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.643891  6468 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:23.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:23.654498  6404 tablet.cc:2369] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:44:23.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.660977  5399 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cafd375e1de4fac89129b600102f54f (127.4.233.194). New cstate: current_term: 1 leader_uuid: "3cafd375e1de4fac89129b600102f54f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: UNKNOWN } } }
19:44:23.740 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 4113d4b6754643fe85f46954d45a80d7
19:44:23.766 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:23.775 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299991, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:23.777 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [12ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=null) after lookup
19:44:23.790 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:23.792 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] Successfully connected to peer
19:44:23.857 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xdb570992, L:/127.0.0.1:33546 - R:/127.4.233.194:40309] using TOKEN/null
19:44:23.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.880491  6296 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Refusing update from remote peer 3cafd375e1de4fac89129b600102f54f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:44:23.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.882500  6478 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
19:44:23.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.888248  5760 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Refusing update from remote peer 3cafd375e1de4fac89129b600102f54f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:44:23.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.890484  6478 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
19:44:23.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.917616  6491 mvcc.cc:205] Tried to move back new op lower bound from 7317411692023652352 to 7317411691094233088. Current Snapshot: MvccSnapshot[applied={T|T < 7317411692023652352}]
19:44:23.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:23.921598  6494 mvcc.cc:205] Tried to move back new op lower bound from 7317411692023652352 to 7317411691094233088. Current Snapshot: MvccSnapshot[applied={T|T < 7317411692023652352}]
19:44:24.096 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:44:24.099 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.194:40309
19:44:24.101 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:44:24.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:24.102057  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 5873
19:44:24.160 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:24.163 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] handling channelInactive
19:44:24.164 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state READY due to: connection closed
19:44:24.165 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: connection closed
19:44:24.175 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:24.175 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:24.177 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test)) after lookup
19:44:24.178 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:24.180 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.180 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.180 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.195 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:24.195 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:24.196 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329))) after lookup
19:44:24.197 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:24.198 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.198 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.198 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.215 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:24.216 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:24.217 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=55), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [35ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [37ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [48ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [53ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313))) after lookup
19:44:24.218 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:24.220 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.221 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.221 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.315 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:24.316 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:24.317 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=155), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [35ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [37ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [48ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [53ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [60ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [60ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [153ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855))) after lookup
19:44:24.318 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:24.319 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.320 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.320 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.796 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:24.797 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:24.798 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=636), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [35ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [37ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [48ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [53ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [60ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [60ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [153ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [156ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [159ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [159ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [628ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [634ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633))) after lookup
19:44:24.799 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:24.801 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.801 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:24.802 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:25.096 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:25.097 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:25.098 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=936), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [35ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [37ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [48ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [53ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [60ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [60ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [153ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [156ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [159ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [159ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [628ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [634ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [637ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [640ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [641ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [928ms] refreshing cache from master, [929ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [934ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633) -> (continuation of Deferred@1543573496 after retry RPC@124366091), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633) -> (continuation of Deferred@1543573496 after retry RPC@124366091))) after lookup
19:44:25.099 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:25.101 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:25.101 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:25.102 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:25.396 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:25.397 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:25.398 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1236), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [35ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [37ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [48ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [53ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [60ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [60ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [153ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [156ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [159ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [159ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [628ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [634ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [637ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [640ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [641ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [928ms] refreshing cache from master, [929ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [934ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [938ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [940ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [941ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [1229ms] refreshing cache from master, [1229ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [1234ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633) -> (continuation of Deferred@1543573496 after retry RPC@124366091) -> (continuation of Deferred@1161626047 after retry RPC@808856877), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633) -> (continuation of Deferred@1543573496 after retry RPC@124366091) -> (continuation of Deferred@1161626047 after retry RPC@808856877))) after lookup
19:44:25.399 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:25.402 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:25.402 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:25.402 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:25.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.527096  6509 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:25.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.527726  6509 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:25.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.530675  6510 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:25.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.531301  6509 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:25.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.531199  6510 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:25.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:25.538281  5641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:25.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:25.543569  6190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:25.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.547708  6510 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:25.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:25.550030  5641 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:25.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:25.555692  6190 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:25.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.558544  5760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a" is_pre_election: true
19:44:25.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.559325  5760 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd83ec522d79445cb7c23e85a2486c50 in term 1.
19:44:25.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.560556  6190 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:44:25.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.561339  6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50" is_pre_election: true
19:44:25.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.561523  6510 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Leader pre-election won for term 2
19:44:25.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.562038  6296 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 1.
19:44:25.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.562389  6510 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:25.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.562778  6510 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 1 FOLLOWER]: Advancing to term 2
19:44:25.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.563333  5641 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:44:25.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.564216  6509 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Leader pre-election won for term 2
19:44:25.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.564610  6509 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:25.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.564968  6509 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 1 FOLLOWER]: Advancing to term 2
19:44:25.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.570631  6510 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:25.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.573426  6510 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 2 election: Requested vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:25.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.573431  6509 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:25.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.575302  5760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:25.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.575572  6509 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 2 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:25.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.576210  5760 raft_consensus.cc:2379] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fd83ec522d79445cb7c23e85a2486c50 in current term 2: Already voted for candidate e093e47eceec459aa9a5b4c4f354183a in this term.
19:44:25.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.576722  6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:25.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.577831  6296 raft_consensus.cc:2379] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e093e47eceec459aa9a5b4c4f354183a in current term 2: Already voted for candidate fd83ec522d79445cb7c23e85a2486c50 in this term.
19:44:25.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:25.579561  6190 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:25.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.580040  6190 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a
19:44:25.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.581025  6510 raft_consensus.cc:2726] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
19:44:25.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:25.584321  5641 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:25.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.584667  5641 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a; no voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50
19:44:25.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:25.585372  6509 raft_consensus.cc:2726] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
19:44:27.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.360983  6525 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:44:27.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.361523  6525 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:27.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.363410  6525 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:27.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.364622  6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50" is_pre_election: true
19:44:27.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.365566  6296 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 2.
19:44:27.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.367622  5641 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 
19:44:27.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.368448  6525 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Leader pre-election won for term 3
19:44:27.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.368808  6525 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
19:44:27.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.369158  6525 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 2 FOLLOWER]: Advancing to term 3
19:44:27.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:27.369993  5641 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.374385  6525 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:27.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.376005  6525 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 3 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:27.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.376989  6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:27.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.377626  6296 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 2 FOLLOWER]: Advancing to term 3
19:44:27.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:27.379541  5641 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:27.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.383100  6296 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 3.
19:44:27.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.384130  5641 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:44:27.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.384865  6525 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 3 FOLLOWER]: Leader election won for term 3
19:44:27.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.386080  6525 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 3 LEADER]: Becoming Leader. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Running, Role: LEADER
19:44:27.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.388404  6525 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:27.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:27.402038  5399 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a reported cstate change: term changed from 1 to 3, leader changed from 3cafd375e1de4fac89129b600102f54f (127.4.233.194) to e093e47eceec459aa9a5b4c4f354183a (127.4.233.193). New cstate: current_term: 3 leader_uuid: "e093e47eceec459aa9a5b4c4f354183a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: HEALTHY } } }
19:44:27.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:27.800659  5641 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:44:28.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:28.000751  6296 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 3 FOLLOWER]: Refusing update from remote peer e093e47eceec459aa9a5b4c4f354183a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
19:44:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:28.002338  6525 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
19:44:29.276 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:29.277 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:29.278 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=16, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=5116), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [13ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [35ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [37ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [37ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [48ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [53ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [60ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [60ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [153ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [156ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [159ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [159ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [628ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [634ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [637ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [640ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [641ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [928ms] refreshing cache from master, [929ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [934ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [938ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [940ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [941ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [1229ms] refreshing cache from master, [1229ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [1234ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [1238ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [1241ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [1241ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [5108ms] refreshing cache from master, [5109ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [5114ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1767617840(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@372f7175 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633) -> (continuation of Deferred@1543573496 after retry RPC@124366091) -> (continuation of Deferred@1161626047 after retry RPC@808856877) -> (continuation of Deferred@138295138 after retry RPC@157297677), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@ac1d5ef -> wakeup thread Time-limited test -> (continuation of Deferred@1581060932 after retry RPC@1515707329) -> (continuation of Deferred@1246641760 after retry RPC@2038990313) -> (continuation of Deferred@1288897905 after retry RPC@1900635855) -> (continuation of Deferred@1189797776 after retry RPC@1570816633) -> (continuation of Deferred@1543573496 after retry RPC@124366091) -> (continuation of Deferred@1161626047 after retry RPC@808856877) -> (continuation of Deferred@138295138 after retry RPC@157297677))) after lookup
19:44:29.280 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:29.281 [DEBUG - kudu-nio-1] (Connection.java:757) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] Successfully connected to peer
19:44:29.311 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x361b8ef3, L:/127.0.0.1:48640 - R:/127.4.233.193:33549] using TOKEN/null
19:44:29.371 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.194:40309
19:44:29.373 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:29.375751  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:44:29.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:29.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:29.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:29.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:29.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:29.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.768704  6589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:29.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.769452  6589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:29.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.769963  6589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:29.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.806263  6589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:29.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.807210  6589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.194
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:29.847530  6589 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:44:29.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:29.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:29.849609  6589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:29.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:29.851908  6589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:29.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.878557  6601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:30.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:30.173579  5641 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:44:30.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:30.772398  5641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111) [suppressed 9 similar messages]
19:44:31.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.878671  6604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:31.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:29.880087  6613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:31.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:31.252700  6605 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1374 milliseconds
19:44:31.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.252862  6589 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:44:31.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.254678  6589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:31.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.258038  6589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:31.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.260249  6589 hybrid_clock.cc:648] HybridClock initialized: now 1786477471260218 us; error 80 us; skew 500 ppm
19:44:31.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.261582  6589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:31.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.271262  6589 webserver.cc:460] Webserver started at http://127.4.233.194:44553/ using document root <none> and password file <none>
19:44:31.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.272680  6589 fs_manager.cc:359] Metadata directory not provided
19:44:31.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.273003  6589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:31.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.285073  6589 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.000s
19:44:31.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.290793  6640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:31.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.292384  6589 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.002s
19:44:31.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.292766  6589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:18 on dist-test-slave-z8x9"
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.295585  6589 fs_report.cc:383] FS layout report
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/data
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:31.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:31.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:31.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.361914  6589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.363982  6589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.364634  6589 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:31.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.368160  6589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:31.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.375972  6665 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:44:31.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.386343  6589 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:44:31.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.386647  6589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s	user 0.003s	sys 0.000s
19:44:31.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.386898  6589 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:44:31.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.393399  6589 ts_tablet_manager.cc:610] Registered 1 tablets
19:44:31.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.393721  6589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s	user 0.007s	sys 0.000s
19:44:31.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.394214  6665 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap starting.
19:44:31.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.458765  6665 log.cc:824] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Log is configured to *not* fsync() on all Append() calls
19:44:31.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.508312  6665 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:44:31.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.509462  6665 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap complete.
19:44:31.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.511210  6665 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent bootstrapping tablet: real 0.118s	user 0.073s	sys 0.038s
19:44:31.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.533970  6665 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:31.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.536842  6665 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Initialized, Role: FOLLOWER
19:44:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.538233  6665 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:31.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.543973  6665 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent starting tablet: real 0.032s	user 0.025s	sys 0.009s
19:44:31.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.603163  6589 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.194:40309
19:44:31.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.603441  6886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.194:40309 every 8 connection(s)
19:44:31.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.606503  6589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:31.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.611207  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 6589
19:44:31.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:31.647 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:44:31.654 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.193:33549
19:44:31.655 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

19:44:31.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.656329  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 5593
19:44:31.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.667158  6889 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.667785  6889 heartbeater.cc:464] Registering TS with master...
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.668968  6889 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.686034  5399 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.691177  6891 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.691628  6891 heartbeater.cc:464] Registering TS with master...
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.691639  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.194:58939
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.692705  6891 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.697629  5255 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:31.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.720198  6892 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:31.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.720664  6892 heartbeater.cc:464] Registering TS with master...
19:44:31.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.721736  6892 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:31.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:31.725582  5111 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:31.764 [DEBUG - kudu-nio-1] (Connection.java:239) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] handling channelInactive
19:44:31.764 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state READY due to: connection closed
19:44:31.765 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:31.766 [DEBUG - Time-limited test] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:31.768 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.769 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.769 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.795 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:31.796 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:31.797 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [3ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [3ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1804975090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@20727499 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21402f66 -> wakeup thread Time-limited test)) after lookup
19:44:31.798 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:31.799 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.799 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.799 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.815 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:31.815 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:31.817 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [3ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [3ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [31ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [33ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [34ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1804975090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@20727499 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21402f66 -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936))) after lookup
19:44:31.818 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:31.820 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.820 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.820 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:31.835 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:31.836 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [3ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [3ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [31ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [33ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [34ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [51ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [54ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [54ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1804975090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@20727499 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21402f66 -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824))) after lookup
19:44:31.837 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:31.839 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.839 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.839 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.976 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:31.976 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:31.977 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=211), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [3ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [3ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [31ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [33ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [34ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [51ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [54ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [54ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [70ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [73ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [73ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [203ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [208ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1804975090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@20727499 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824) -> (continuation of Deferred@453920362 after retry RPC@1355003737), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21402f66 -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824) -> (continuation of Deferred@453920362 after retry RPC@1355003737))) after lookup
19:44:31.978 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:31.980 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.980 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:31.981 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:32.239 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:32.239 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:32.240 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=474), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [3ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [3ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [31ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [33ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [34ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [51ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [54ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [54ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [70ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [73ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [73ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [203ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [208ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [212ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [215ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [215ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [463ms] refreshing cache from master, [464ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [469ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1804975090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@20727499 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824) -> (continuation of Deferred@453920362 after retry RPC@1355003737) -> (continuation of Deferred@17441991 after retry RPC@1287529152), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21402f66 -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824) -> (continuation of Deferred@453920362 after retry RPC@1355003737) -> (continuation of Deferred@17441991 after retry RPC@1287529152))) after lookup
19:44:32.242 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:32.243 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:32.243 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:32.243 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:32.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.703933  6889 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.860190  6913 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:44:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.861080  6913 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.861320  6914 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e093e47eceec459aa9a5b4c4f354183a)
19:44:32.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.862043  6914 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:32.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.863843  6913 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:32.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.864455  6914 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:32.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:32.881271  6190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111) [suppressed 1 similar messages]
19:44:32.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:32.884176  6645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:44:32.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.894364  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "3cafd375e1de4fac89129b600102f54f" is_pre_election: true
19:44:32.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.895557  6802 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd83ec522d79445cb7c23e85a2486c50 in term 1.
19:44:32.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:32.895565  6645 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:44:32.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:32.896123  6190 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:44:32.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.897598  6190 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50; no voters: e093e47eceec459aa9a5b4c4f354183a
19:44:32.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.899112  6914 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 3 FOLLOWER]: Leader pre-election won for term 4
19:44:32.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.899428  6914 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e093e47eceec459aa9a5b4c4f354183a)
19:44:32.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.899845  6914 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 3 FOLLOWER]: Advancing to term 4
19:44:32.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.899892  6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50" is_pre_election: true
19:44:32.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.907096  6914 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:32.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.908445  6296 raft_consensus.cc:2354] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cafd375e1de4fac89129b600102f54f for earlier term 2. Current term is 4.
19:44:32.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.909428  6914 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 4 election: Requested vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:32.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.910604  6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:32.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.911264  6802 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 1 FOLLOWER]: Advancing to term 4
19:44:32.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.914614  6645 leader_election.cc:398] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 2 pre-election: Vote denied by peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997) with higher term. Message: Invalid argument: T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cafd375e1de4fac89129b600102f54f for earlier term 2. Current term is 4.
19:44:32.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.915001  6645 leader_election.cc:401] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
19:44:32.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:32.916056  6190 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:44:32.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.922132  6802 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd83ec522d79445cb7c23e85a2486c50 in term 4.
19:44:32.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.922662  6913 raft_consensus.cc:2726] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997) with higher term. Message: Invalid argument: T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3cafd375e1de4fac89129b600102f54f for earlier term 2. Current term is 4.
19:44:32.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.923509  6190 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50; no voters: e093e47eceec459aa9a5b4c4f354183a
19:44:32.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.924497  6914 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Leader election won for term 4
19:44:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.926721  6914 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 LEADER]: Becoming Leader. State: Replica: fd83ec522d79445cb7c23e85a2486c50, State: Running, Role: LEADER
19:44:32.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.927870  6914 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:32.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:32.939945  5399 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 reported cstate change: term changed from 3 to 4, leader changed from e093e47eceec459aa9a5b4c4f354183a (127.4.233.193) to fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195). New cstate: current_term: 4 leader_uuid: "fd83ec522d79445cb7c23e85a2486c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: UNKNOWN } } }
19:44:33.016 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:33.016 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:33.018 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1251), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [3ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [3ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [31ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [33ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [34ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [48ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [51ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [54ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [54ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [70ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [73ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [73ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [203ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [208ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [212ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [215ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [215ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [463ms] refreshing cache from master, [464ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [469ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [475ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [477ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [477ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [1243ms] refreshing cache from master, [1243ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [1249ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1804975090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@20727499 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824) -> (continuation of Deferred@453920362 after retry RPC@1355003737) -> (continuation of Deferred@17441991 after retry RPC@1287529152) -> (continuation of Deferred@3366213 after retry RPC@1000348430), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21402f66 -> wakeup thread Time-limited test -> (continuation of Deferred@786842621 after retry RPC@134389936) -> (continuation of Deferred@1831560061 after retry RPC@1890666824) -> (continuation of Deferred@453920362 after retry RPC@1355003737) -> (continuation of Deferred@17441991 after retry RPC@1287529152) -> (continuation of Deferred@3366213 after retry RPC@1000348430))) after lookup
19:44:33.019 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:33.021 [DEBUG - kudu-nio-0] (Connection.java:757) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] Successfully connected to peer
19:44:33.050 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x2503473c, L:/127.0.0.1:37562 - R:/127.4.233.195:34997] using TOKEN/null
19:44:33.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:33.063442  6802 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Refusing update from remote peer fd83ec522d79445cb7c23e85a2486c50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
19:44:33.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.063800  6190 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 -> Peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Couldn't send request to peer e093e47eceec459aa9a5b4c4f354183a. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:44:33.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:33.065390  6927 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
19:44:33.143 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.193:33549
19:44:33.144 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

19:44:33.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:33.146226  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:33.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:33.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:33.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:33549
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44387
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:33.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:33.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.545707  6969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:33.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.546321  6969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:33.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.546788  6969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:33.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.585296  6969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:33.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.586272  6969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.193
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:33.648381  6969 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:33549
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44387
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:33.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:33.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:33.650372  6969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:33.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:33.652904  6969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:33.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.681084  6987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:34.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.682502  6985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:34.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:33.682070  6984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:34.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:34.965711  6986 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:44:34.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.965902  6969 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:44:34.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.967904  6969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:34.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.970923  6969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:34.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.972488  6969 hybrid_clock.cc:648] HybridClock initialized: now 1786477474972418 us; error 114 us; skew 500 ppm
19:44:34.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.973592  6969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.981734  6969 webserver.cc:460] Webserver started at http://127.4.233.193:44387/ using document root <none> and password file <none>
19:44:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.982937  6969 fs_manager.cc:359] Metadata directory not provided
19:44:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.983209  6969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:34.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.993242  6969 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.003s	sys 0.003s
19:44:34.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:34.999076  7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:35.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.000633  6969 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.001204  6969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:16 on dist-test-slave-z8x9"
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.004148  6969 fs_report.cc:383] FS layout report
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/data
19:44:35.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:35.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:35.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.051330  6969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.053284  6969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.053885  6969 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:35.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.057004  6969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:35.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.064323  7023 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:44:35.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.074363  6969 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:44:35.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.074728  6969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.000s
19:44:35.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.075093  6969 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:44:35.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.081851  6969 ts_tablet_manager.cc:610] Registered 1 tablets
19:44:35.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.082093  6969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s	user 0.007s	sys 0.000s
19:44:35.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.082545  7023 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap starting.
19:44:35.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.151298  7023 log.cc:824] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Log is configured to *not* fsync() on all Append() calls
19:44:35.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.218648  7023 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:44:35.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.221410  7023 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap complete.
19:44:35.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.224429  7023 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent bootstrapping tablet: real 0.142s	user 0.092s	sys 0.040s
19:44:35.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.253273  7023 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:35.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.257814  7023 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Initialized, Role: FOLLOWER
19:44:35.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.259253  7023 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:35.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.283660  7023 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent starting tablet: real 0.059s	user 0.048s	sys 0.006s
19:44:35.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.353067  6969 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.193:33549
19:44:35.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.353382  7221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.193:33549 every 8 connection(s)
19:44:35.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.356475  6969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:35.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.357317  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 6969
19:44:35.358 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:35.393 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:44:35.396 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.195:34997
19:44:35.397 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

19:44:35.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.398042  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 6148
19:44:35.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.438288  7223 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:35.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.438944  7223 heartbeater.cc:464] Registering TS with master...
19:44:35.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.440580  7223 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:35.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.443384  7241 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:35.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.443794  7241 heartbeater.cc:464] Registering TS with master...
19:44:35.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.444465  7226 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:35.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.444823  7241 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:35.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.444846  7226 heartbeater.cc:464] Registering TS with master...
19:44:35.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.446000  7226 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:35.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.446596  5399 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:35.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.449002  5111 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:35.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.450214  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.193:38223
19:44:35.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:35.452484  5255 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:35.472 [DEBUG - kudu-nio-0] (Connection.java:239) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] handling channelInactive
19:44:35.473 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state READY due to: connection closed
19:44:35.473 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:35.474 [DEBUG - Time-limited test] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:35.476 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.476 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.477 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.494 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:35.495 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:35.495 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [20ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@2055094687(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@49a8c1f1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7df186c2 -> wakeup thread Time-limited test)) after lookup
19:44:35.496 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:35.499 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.499 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.499 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.514 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:35.515 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:35.515 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [20ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [22ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [25ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [25ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@2055094687(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@49a8c1f1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7df186c2 -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087))) after lookup
19:44:35.516 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:35.517 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.517 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.517 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.554 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:35.555 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:35.555 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [20ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [22ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [25ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [25ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [41ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [43ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [43ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [80ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@2055094687(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@49a8c1f1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7df186c2 -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254))) after lookup
19:44:35.556 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:35.559 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.559 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.559 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.655 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:35.655 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:35.656 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=182), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [20ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [22ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [25ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [25ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [41ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [43ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [43ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [80ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [82ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [85ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [85ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [175ms] refreshing cache from master, [175ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [180ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@2055094687(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@49a8c1f1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7df186c2 -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991))) after lookup
19:44:35.657 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:35.658 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.658 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.658 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.815 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:35.816 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:35.817 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=342), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [20ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [22ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [25ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [25ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [41ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [43ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [43ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [80ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [82ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [85ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [85ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [175ms] refreshing cache from master, [175ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [180ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [182ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [184ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [185ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [336ms] refreshing cache from master, [336ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [340ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@2055094687(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@49a8c1f1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991) -> (continuation of Deferred@2047989409 after retry RPC@1779502859), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7df186c2 -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991) -> (continuation of Deferred@2047989409 after retry RPC@1779502859))) after lookup
19:44:35.817 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:35.819 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.819 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.820 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.895 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:35.895 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:35.896 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=421), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [20ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [22ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [25ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [25ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [41ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [43ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [43ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [80ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [82ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [85ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [85ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [175ms] refreshing cache from master, [175ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [180ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [182ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [184ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [185ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [336ms] refreshing cache from master, [336ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [340ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [343ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [346ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [346ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [415ms] refreshing cache from master, [415ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [420ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@2055094687(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@49a8c1f1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991) -> (continuation of Deferred@2047989409 after retry RPC@1779502859) -> (continuation of Deferred@2001140143 after retry RPC@74302733), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7df186c2 -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991) -> (continuation of Deferred@2047989409 after retry RPC@1779502859) -> (continuation of Deferred@2001140143 after retry RPC@74302733))) after lookup
19:44:35.897 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:35.898 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.898 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:35.898 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:36.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.445031  7256 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fd83ec522d79445cb7c23e85a2486c50)
19:44:36.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.445637  7256 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:36.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.447642  7256 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:36.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:36.455142  6645 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:44:36.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.457297  7223 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:36.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.465546  7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a" is_pre_election: true
19:44:36.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.466723  7136 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 3.
19:44:36.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.468281  6645 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a; no voters: fd83ec522d79445cb7c23e85a2486c50
19:44:36.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.469115  7256 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Leader pre-election won for term 5
19:44:36.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.469543  7256 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Starting leader election (detected failure of leader fd83ec522d79445cb7c23e85a2486c50)
19:44:36.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.469816  7256 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 4 FOLLOWER]: Advancing to term 5
19:44:36.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.475163  7256 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:36.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.477198  7256 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 5 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:36.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.479136  7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:36.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.479875  7136 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 3 FOLLOWER]: Advancing to term 5
19:44:36.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:36.480765  6645 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:44:36.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.488965  7136 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 5.
19:44:36.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.490221  6645 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a; no voters: fd83ec522d79445cb7c23e85a2486c50
19:44:36.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.491034  7256 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 5 FOLLOWER]: Leader election won for term 5
19:44:36.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.492440  7256 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 5 LEADER]: Becoming Leader. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Running, Role: LEADER
19:44:36.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.493335  7256 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:36.502930  5399 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f reported cstate change: term changed from 4 to 5, leader changed from fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195) to 3cafd375e1de4fac89129b600102f54f (127.4.233.194). New cstate: current_term: 5 leader_uuid: "3cafd375e1de4fac89129b600102f54f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: UNKNOWN } } }
19:44:37.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:37.046806  7136 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 5 FOLLOWER]: Refusing update from remote peer 3cafd375e1de4fac89129b600102f54f: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
19:44:37.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:37.048779  7296 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.001s
19:44:37.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:37.054484  6645 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f -> Peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Couldn't send request to peer fd83ec522d79445cb7c23e85a2486c50. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:44:38.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.122076  6645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111) [suppressed 4 similar messages]
19:44:38.295 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:38.295 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:38.296 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2822), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [20ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [22ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [25ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [25ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [40ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [41ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [43ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [43ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [80ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [82ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [85ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [85ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [175ms] refreshing cache from master, [175ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [180ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [182ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [184ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [185ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [336ms] refreshing cache from master, [336ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [340ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [343ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [346ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [346ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [415ms] refreshing cache from master, [415ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [420ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [422ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [424ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [424ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [2815ms] refreshing cache from master, [2815ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [2820ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@2055094687(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@49a8c1f1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991) -> (continuation of Deferred@2047989409 after retry RPC@1779502859) -> (continuation of Deferred@2001140143 after retry RPC@74302733) -> (continuation of Deferred@244266933 after retry RPC@2066056898), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7df186c2 -> wakeup thread Time-limited test -> (continuation of Deferred@1840322069 after retry RPC@470244087) -> (continuation of Deferred@1930713612 after retry RPC@832429254) -> (continuation of Deferred@2011044206 after retry RPC@679116991) -> (continuation of Deferred@2047989409 after retry RPC@1779502859) -> (continuation of Deferred@2001140143 after retry RPC@74302733) -> (continuation of Deferred@244266933 after retry RPC@2066056898))) after lookup
19:44:38.297 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:38.300 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] Successfully connected to peer
19:44:38.326 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x607f2a96, L:/127.0.0.1:39276 - R:/127.4.233.194:40309] using TOKEN/null
19:44:38.405 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.195:34997
19:44:38.405 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

19:44:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:38.408156  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:44:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:34997
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45763
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:38.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:38.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.823904  7344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:38.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.824642  7344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:38.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.825464  7344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:38.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.862385  7344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:38.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.863332  7344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.195
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:38.907899  7344 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:34997
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45763
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:38.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:38.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:38.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:38.910000  7344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:38.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:38.912441  7344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:38.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.938153  7355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:39.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:39.512946  6645 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f -> Peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Couldn't send request to peer fd83ec522d79445cb7c23e85a2486c50. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:44:40.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.939133  7362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:40.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:38.939982  7344 server_base.cc:1034] running on GCE node
19:44:40.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:38.938232  7357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:40.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.228415  7344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:40.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.232025  7344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:40.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.235926  7344 hybrid_clock.cc:648] HybridClock initialized: now 1786477480236181 us; error 510 us; skew 500 ppm
19:44:40.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.237128  7344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:40.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.255028  7344 webserver.cc:460] Webserver started at http://127.4.233.195:45763/ using document root <none> and password file <none>
19:44:40.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.256337  7344 fs_manager.cc:359] Metadata directory not provided
19:44:40.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.256659  7344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:40.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.267378  7344 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.008s	sys 0.000s
19:44:40.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.273115  7381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.274482  7344 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.002s
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.274915  7344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:21 on dist-test-slave-z8x9"
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.277902  7344 fs_report.cc:383] FS layout report
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/data
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:40.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:40.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.326841  7344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:40.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.328642  7344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:40.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.329273  7344 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:40.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.332438  7344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:40.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.340210  7396 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:44:40.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.352370  7344 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:44:40.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.352715  7344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s	user 0.002s	sys 0.001s
19:44:40.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.353080  7344 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:44:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.362218  7344 ts_tablet_manager.cc:610] Registered 1 tablets
19:44:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.362555  7344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s	user 0.009s	sys 0.000s
19:44:40.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.363507  7396 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap starting.
19:44:40.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.433657  7396 log.cc:824] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Log is configured to *not* fsync() on all Append() calls
19:44:40.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.495554  7396 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:44:40.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.496618  7396 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap complete.
19:44:40.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.498751  7396 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent bootstrapping tablet: real 0.136s	user 0.077s	sys 0.053s
19:44:40.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.524374  7396 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:40.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.528281  7396 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd83ec522d79445cb7c23e85a2486c50, State: Initialized, Role: FOLLOWER
19:44:40.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.529474  7396 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:40.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.534468  7396 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent starting tablet: real 0.035s	user 0.030s	sys 0.006s
19:44:40.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.590759  7344 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.195:34997
19:44:40.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.590982  7602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.195:34997 every 8 connection(s)
19:44:40.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.594031  7344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:40.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.608633  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 7344
19:44:40.610 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:40.672 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:44:40.704 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.194:40309
19:44:40.705 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:44:40.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.705771  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 6589
19:44:40.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.739753  7608 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.740163  7605 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.740221  7603 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.740432  7608 heartbeater.cc:464] Registering TS with master...
19:44:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.740545  7605 heartbeater.cc:464] Registering TS with master...
19:44:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.740659  7603 heartbeater.cc:464] Registering TS with master...
19:44:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.742221  7603 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.742360  7608 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.744058  7605 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.747962  5397 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:40.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.751538  5256 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:40.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.752241  5397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.195:58379
19:44:40.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:40.756316  5111 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:40.776 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] handling channelInactive
19:44:40.777 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state READY due to: connection closed
19:44:40.777 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:40.778 [DEBUG - Time-limited test] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:40.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:40.776603  5169 debug-util.cc:398] Leaking SignalData structure 0x7b08000a3320 after lost signal to thread 5056
19:44:40.782 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.782 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.782 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.796 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:40.797 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:40.797 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [4ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@151962080(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1d02a594 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5345c8ff -> wakeup thread Time-limited test)) after lookup
19:44:40.799 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:40.802 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.802 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.802 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.823 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:40.824 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:40.824 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [4ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [20ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [24ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [24ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@151962080(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1d02a594 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5345c8ff -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459))) after lookup
19:44:40.825 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:40.826 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.826 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.827 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.874 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:40.875 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:40.875 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=98), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [4ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [20ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [24ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [24ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [47ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [49ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [49ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [91ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [96ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@151962080(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1d02a594 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5345c8ff -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849))) after lookup
19:44:40.877 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:40.879 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.879 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.879 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.918 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:40.919 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:40.919 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=141), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [4ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [20ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [24ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [24ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [47ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [49ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [49ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [91ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [96ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [98ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [101ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [102ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [131ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [139ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@151962080(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1d02a594 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5345c8ff -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416))) after lookup
19:44:40.921 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:40.926 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.926 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.926 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.957 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:40.957 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:40.958 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=180), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [4ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [20ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [24ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [24ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [47ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [49ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [49ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [91ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [96ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [98ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [101ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [102ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [131ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [139ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [142ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [148ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [171ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [178ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@151962080(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1d02a594 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416) -> (continuation of Deferred@1223893664 after retry RPC@1523853485), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5345c8ff -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416) -> (continuation of Deferred@1223893664 after retry RPC@1523853485))) after lookup
19:44:40.963 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:40.966 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.966 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:40.966 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:41.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:41.335 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:41.336 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=558), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [4ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [20ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [24ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [24ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [47ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [49ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [49ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [91ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [96ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [98ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [101ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [102ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [131ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [139ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [142ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [148ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [171ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [178ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [181ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [188ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [188ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [551ms] refreshing cache from master, [552ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [556ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@151962080(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1d02a594 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416) -> (continuation of Deferred@1223893664 after retry RPC@1523853485) -> (continuation of Deferred@1794312293 after retry RPC@286901462), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5345c8ff -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416) -> (continuation of Deferred@1223893664 after retry RPC@1523853485) -> (continuation of Deferred@1794312293 after retry RPC@286901462))) after lookup
19:44:41.337 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:41.338 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:41.339 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:41.339 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:41.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.758483  7603 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:41.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.849084  7629 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:41.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.849613  7629 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:41.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.852442  7629 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:41.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:41.857024  7019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:41.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:41.860843  7019 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:41.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.867906  7507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50" is_pre_election: true
19:44:41.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.868791  7507 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 4.
19:44:41.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.870123  7019 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:44:41.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.870872  7629 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 5 FOLLOWER]: Leader pre-election won for term 6
19:44:41.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.871152  7629 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:41.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.871393  7629 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 5 FOLLOWER]: Advancing to term 6
19:44:41.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.876477  7629 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.878204  7629 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 6 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:41.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.879331  7507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.879856  7507 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 4 FOLLOWER]: Advancing to term 6
19:44:41.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:41.882228  7019 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:41.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.888381  7507 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 6.
19:44:41.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.889566  7019 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:44:41.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.890301  7629 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 6 FOLLOWER]: Leader election won for term 6
19:44:41.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.892373  7629 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 6 LEADER]: Becoming Leader. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Running, Role: LEADER
19:44:41.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.893687  7629 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:41.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:41.903571  5397 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a reported cstate change: term changed from 5 to 6, leader changed from 3cafd375e1de4fac89129b600102f54f (127.4.233.194) to e093e47eceec459aa9a5b4c4f354183a (127.4.233.193). New cstate: current_term: 6 leader_uuid: "e093e47eceec459aa9a5b4c4f354183a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: HEALTHY } } }
19:44:41.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:41.924551  5169 debug-util.cc:398] Leaking SignalData structure 0x7b08000a5600 after lost signal to thread 5056
19:44:41.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:41.925468  5169 debug-util.cc:398] Leaking SignalData structure 0x7b08000a2ce0 after lost signal to thread 5174
19:44:42.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:42.304426  7019 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:44:42.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:42.441082  7507 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 6 FOLLOWER]: Refusing update from remote peer e093e47eceec459aa9a5b4c4f354183a: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
19:44:42.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:42.442891  7629 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
19:44:42.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:42.835 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:42.836 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2058), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [4ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [4ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [20ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [24ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [24ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [45ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [47ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [49ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [49ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [91ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [96ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [98ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [101ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [102ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [131ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [139ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [142ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [148ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [148ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [171ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [178ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [181ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [188ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [188ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [551ms] refreshing cache from master, [552ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [556ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [559ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [561ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [561ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2051ms] refreshing cache from master, [2052ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [2056ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@151962080(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1d02a594 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416) -> (continuation of Deferred@1223893664 after retry RPC@1523853485) -> (continuation of Deferred@1794312293 after retry RPC@286901462) -> (continuation of Deferred@2053987895 after retry RPC@1133604785), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5345c8ff -> wakeup thread Time-limited test -> (continuation of Deferred@428699789 after retry RPC@1685209459) -> (continuation of Deferred@1379076929 after retry RPC@1762165849) -> (continuation of Deferred@674600643 after retry RPC@95614416) -> (continuation of Deferred@1223893664 after retry RPC@1523853485) -> (continuation of Deferred@1794312293 after retry RPC@286901462) -> (continuation of Deferred@2053987895 after retry RPC@1133604785))) after lookup
19:44:42.837 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:42.839 [DEBUG - kudu-nio-0] (Connection.java:757) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] Successfully connected to peer
19:44:42.894 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x7015ae2e, L:/127.0.0.1:45784 - R:/127.4.233.193:33549] using TOKEN/null
19:44:42.954 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.194:40309
19:44:42.954 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:44:42.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:42.957471  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:44:42.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:42.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:42.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:42.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:42.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.346590  7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.347229  7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.347721  7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:43.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.386844  7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:43.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.387836  7684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.194
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:43.429473  7684 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:43.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:43.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:43.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:43.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:43.431110  7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:43.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:43.433452  7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:43.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.458173  7693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:44.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.458411  7697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:44.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:43.458606  7694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:44.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:43.461790  7684 server_base.cc:1034] running on GCE node
19:44:44.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:44.744464  7019 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:44:44.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.746232  7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:44.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.748994  7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:44.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.750484  7684 hybrid_clock.cc:648] HybridClock initialized: now 1786477484750430 us; error 75 us; skew 500 ppm
19:44:44.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.751602  7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:44.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.760506  7684 webserver.cc:460] Webserver started at http://127.4.233.194:44553/ using document root <none> and password file <none>
19:44:44.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.761857  7684 fs_manager.cc:359] Metadata directory not provided
19:44:44.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.762111  7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:44.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.771994  7684 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.004s
19:44:44.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.778262  7723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:44.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.779824  7684 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
19:44:44.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.780273  7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:44.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:44.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:18 on dist-test-slave-z8x9"
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.783394  7684 fs_report.cc:383] FS layout report
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/data
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:44.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.828285  7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:44.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.830418  7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:44.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.831058  7684 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:44.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.834156  7684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:44.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.841351  7733 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:44:44.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.851063  7684 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:44:44.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.851430  7684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.001s
19:44:44.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.851703  7684 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:44:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.857650  7684 ts_tablet_manager.cc:610] Registered 1 tablets
19:44:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.857867  7684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.003s	sys 0.003s
19:44:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.858505  7733 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap starting.
19:44:44.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:44.929912  7733 log.cc:824] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Log is configured to *not* fsync() on all Append() calls
19:44:45.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.015712  7733 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:44:45.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.017650  7733 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap complete.
19:44:45.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.020361  7733 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent bootstrapping tablet: real 0.162s	user 0.113s	sys 0.039s
19:44:45.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.058311  7733 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:45.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.064016  7733 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Initialized, Role: FOLLOWER
19:44:45.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.065997  7733 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:45.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.072279  7733 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent starting tablet: real 0.051s	user 0.038s	sys 0.009s
19:44:45.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.123056  7684 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.194:40309
19:44:45.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.123330  7949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.194:40309 every 8 connection(s)
19:44:45.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.126248  7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.128324  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 7684
19:44:45.129 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:45.139 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
19:44:45.140 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
19:44:45.177 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:44:45.185 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.193:33549
19:44:45.186 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

19:44:45.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.186764  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 6969
19:44:45.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.237815  7959 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:45.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.238481  7959 heartbeater.cc:464] Registering TS with master...
19:44:45.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.240065  7959 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:45.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.247370  5111 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:45.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.251744  7956 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:45.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.252213  7956 heartbeater.cc:464] Registering TS with master...
19:44:45.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.253351  7956 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:45.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.256316  7958 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:45.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.256737  7958 heartbeater.cc:464] Registering TS with master...
19:44:45.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.257828  7958 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:45.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.257863  5399 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:45.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.263041  5256 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:45.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:45.266131  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.194:46031
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:45.275107  7976 negotiation.cc:337] Failed RPC negotiation. Trace:
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.184745 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.233.193:44569 (local address 127.4.233.194:40309)
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.240791 (+ 56046us) server_negotiation.cc:206] Beginning negotiation
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.240821 (+    30us) server_negotiation.cc:399] Waiting for connection header
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.240891 (+    70us) server_negotiation.cc:407] Connection header received
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.241115 (+   224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.241144 (+    29us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.241659 (+   515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.241913 (+   254us) server_negotiation.cc:226] Negotiated authn=SASL
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:45.273680 (+ 31767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.233.193:44569: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
19:44:45.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":53186,"thread_start_us":64274,"threads_started":1}
19:44:45.276 [DEBUG - kudu-nio-0] (Connection.java:239) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] handling channelInactive
19:44:45.276 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state READY due to: connection closed
19:44:45.279 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:45.280 [DEBUG - Time-limited test] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:45.283 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.283 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.283 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.294 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:45.294 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:45.295 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [4ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [4ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1362266195(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@485779c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@22057f56 -> wakeup thread Time-limited test)) after lookup
19:44:45.296 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:45.298 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.298 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.298 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.314 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:45.314 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:45.315 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [4ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [4ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [19ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [19ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1362266195(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@485779c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@22057f56 -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154))) after lookup
19:44:45.316 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:45.318 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.318 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.318 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:45.336 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:45.336 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [4ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [4ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [19ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [19ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [37ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [39ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [39ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1362266195(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@485779c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@22057f56 -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468))) after lookup
19:44:45.337 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:45.340 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.340 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.340 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.415 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:45.415 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:45.416 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=136), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [4ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [4ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [19ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [19ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [37ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [39ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [39ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [58ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [61ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [61ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [130ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [135ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1362266195(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@485779c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468) -> (continuation of Deferred@712571342 after retry RPC@1091108685), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@22057f56 -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468) -> (continuation of Deferred@712571342 after retry RPC@1091108685))) after lookup
19:44:45.418 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:45.419 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.419 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.420 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.435 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:45.436 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:45.437 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=157), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [4ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [4ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [19ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [19ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [37ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [39ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [39ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [58ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [61ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [61ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [130ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [135ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [139ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [140ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [141ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [151ms] refreshing cache from master, [151ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [156ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1362266195(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@485779c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468) -> (continuation of Deferred@712571342 after retry RPC@1091108685) -> (continuation of Deferred@1132191295 after retry RPC@63018335), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@22057f56 -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468) -> (continuation of Deferred@712571342 after retry RPC@1091108685) -> (continuation of Deferred@1132191295 after retry RPC@63018335))) after lookup
19:44:45.439 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:45.441 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.441 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:45.441 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:44:46.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.273586  7956 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:46.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.530452  7980 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e093e47eceec459aa9a5b4c4f354183a)
19:44:46.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.531052  7980 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:46.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.533787  7980 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:46.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:46.538275  7390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:44:46.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:46.541991  7390 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:44:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.548291  7846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "3cafd375e1de4fac89129b600102f54f" is_pre_election: true
19:44:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.549121  7846 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd83ec522d79445cb7c23e85a2486c50 in term 5.
19:44:46.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.550669  7390 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50; no voters: e093e47eceec459aa9a5b4c4f354183a
19:44:46.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.551488  7980 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 6 FOLLOWER]: Leader pre-election won for term 7
19:44:46.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.551833  7980 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e093e47eceec459aa9a5b4c4f354183a)
19:44:46.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.552069  7980 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 6 FOLLOWER]: Advancing to term 7
19:44:46.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.557092  7980 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:46.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.558888  7980 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 7 election: Requested vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:46.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.559892  7846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:46.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.560580  7846 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 5 FOLLOWER]: Advancing to term 7
19:44:46.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:46.562983  7390 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:44:46.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.569082  7846 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd83ec522d79445cb7c23e85a2486c50 in term 7.
19:44:46.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.570389  7390 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50; no voters: e093e47eceec459aa9a5b4c4f354183a
19:44:46.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.571193  7980 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 7 FOLLOWER]: Leader election won for term 7
19:44:46.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.573359  7980 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 7 LEADER]: Becoming Leader. State: Replica: fd83ec522d79445cb7c23e85a2486c50, State: Running, Role: LEADER
19:44:46.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.574786  7980 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:46.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:46.585204  5399 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 reported cstate change: term changed from 6 to 7, leader changed from e093e47eceec459aa9a5b4c4f354183a (127.4.233.193) to fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195). New cstate: current_term: 7 leader_uuid: "fd83ec522d79445cb7c23e85a2486c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: UNKNOWN } } }
19:44:47.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:47.002694  7846 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 7 FOLLOWER]: Refusing update from remote peer fd83ec522d79445cb7c23e85a2486c50: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
19:44:47.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:47.004464  7980 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
19:44:47.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:47.041538  7390 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 -> Peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Couldn't send request to peer e093e47eceec459aa9a5b4c4f354183a. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:44:47.434 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:47.435 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:47.436 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2156), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [4ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [4ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [19ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [19ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [34ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [37ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [39ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [39ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [58ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [61ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [61ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [130ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [135ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [139ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [140ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [141ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [151ms] refreshing cache from master, [151ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [156ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [159ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [162ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [163ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2150ms] refreshing cache from master, [2150ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [2155ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1362266195(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@485779c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468) -> (continuation of Deferred@712571342 after retry RPC@1091108685) -> (continuation of Deferred@1132191295 after retry RPC@63018335) -> (continuation of Deferred@1000113957 after retry RPC@888702564), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@22057f56 -> wakeup thread Time-limited test -> (continuation of Deferred@104771412 after retry RPC@264806154) -> (continuation of Deferred@711379716 after retry RPC@576272468) -> (continuation of Deferred@712571342 after retry RPC@1091108685) -> (continuation of Deferred@1132191295 after retry RPC@63018335) -> (continuation of Deferred@1000113957 after retry RPC@888702564))) after lookup
19:44:47.437 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:47.438 [DEBUG - kudu-nio-1] (Connection.java:757) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] Successfully connected to peer
19:44:47.469 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x920ad854, L:/127.0.0.1:42210 - R:/127.4.233.195:34997] using TOKEN/null
19:44:47.524 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.193:33549
19:44:47.525 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:47.527626  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:33549
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44387
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:44:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:47.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:47.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:47.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:47.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:47.894631  8034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:47.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:47.895237  8034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:47.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:47.895718  8034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:47.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:47.931440  8034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:47.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:47.932389  8034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.193
19:44:47.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:47.974232  8034 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:33549
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44387
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:47.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:47.975800  8034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:47.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:47.978113  8034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:48.001030  8047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:49.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:49.251433  7390 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 -> Peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Couldn't send request to peer e093e47eceec459aa9a5b4c4f354183a. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:44:49.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:48.002732  8043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:49.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:48.001057  8052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:49.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:49.397385  8049 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1392 milliseconds
19:44:49.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.400354  8034 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:44:49.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.402379  8034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:49.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.405597  8034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:49.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.407006  8034 hybrid_clock.cc:648] HybridClock initialized: now 1786477489406917 us; error 94 us; skew 500 ppm
19:44:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.407908  8034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:49.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.415728  8034 webserver.cc:460] Webserver started at http://127.4.233.193:44387/ using document root <none> and password file <none>
19:44:49.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.416793  8034 fs_manager.cc:359] Metadata directory not provided
19:44:49.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.417011  8034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.426411  8034 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.008s	sys 0.000s
19:44:49.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.431710  8079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:49.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.432921  8034 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
19:44:49.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.433363  8034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:49.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:16 on dist-test-slave-z8x9"
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.436856  8034 fs_report.cc:383] FS layout report
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/data
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:49.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:49.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.497607  8034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:49.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.499682  8034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:49.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.500324  8034 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:49.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.503659  8034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:49.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.510937  8089 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:44:49.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.521729  8034 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:44:49.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.521983  8034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s	user 0.000s	sys 0.002s
19:44:49.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.522233  8034 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:44:49.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.528294  8034 ts_tablet_manager.cc:610] Registered 1 tablets
19:44:49.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.528519  8034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.002s	sys 0.001s
19:44:49.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.529003  8089 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap starting.
19:44:49.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.601432  8089 log.cc:824] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Log is configured to *not* fsync() on all Append() calls
19:44:49.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.683091  8089 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:44:49.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.684445  8089 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap complete.
19:44:49.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.686676  8089 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent bootstrapping tablet: real 0.158s	user 0.112s	sys 0.040s
19:44:49.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.710489  8089 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:49.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.713510  8089 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Initialized, Role: FOLLOWER
19:44:49.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.714871  8089 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.720906  8089 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent starting tablet: real 0.034s	user 0.029s	sys 0.004s
19:44:49.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.764964  8034 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.193:33549
19:44:49.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.765293  8317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.193:33549 every 8 connection(s)
19:44:49.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.768414  8034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:44:49.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.774868  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 8034
19:44:49.777 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:49.821 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:44:49.822 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.195:34997
19:44:49.823 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

19:44:49.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.823946  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 7344
19:44:49.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.859427  8319 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.860239  8319 heartbeater.cc:464] Registering TS with master...
19:44:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.861713  8319 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.863293  8318 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:49.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.863864  8318 heartbeater.cc:464] Registering TS with master...
19:44:49.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.865233  8318 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.868310  5256 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.869532  5399 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:49.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.873344  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.193:34831
19:44:49.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.880977  8320 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:49.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.881453  8320 heartbeater.cc:464] Registering TS with master...
19:44:49.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.882179  8320 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:49.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:49.886126  5112 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:49.911 [DEBUG - kudu-nio-1] (Connection.java:239) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] handling channelInactive
19:44:49.911 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state READY due to: connection closed
19:44:49.912 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:49.912818  8332 negotiation.cc:337] Failed RPC negotiation. Trace:
19:44:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:49.807192 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.233.195:51629 (local address 127.4.233.193:33549)
19:44:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:49.828643 (+ 21451us) server_negotiation.cc:206] Beginning negotiation
19:44:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:49.828698 (+    55us) server_negotiation.cc:399] Waiting for connection header
19:44:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:49.828794 (+    96us) server_negotiation.cc:407] Connection header received
19:44:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 19:44:49.910648 (+ 81854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.233.195:51629: BlockingRecv error: recv got EOF from 127.4.233.195:51629 (error 108)
19:44:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":20618,"thread_start_us":21545,"threads_started":1}
19:44:49.913 [DEBUG - Time-limited test] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:49.915 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.915 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.915 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.935 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:49.936 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:49.936 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [22ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1721700812(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1a51810a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76862469 -> wakeup thread Time-limited test)) after lookup
19:44:49.937 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:49.938 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.938 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.939 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.955 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:49.955 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:49.956 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [22ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [24ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [27ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [27ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [42ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1721700812(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1a51810a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76862469 -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050))) after lookup
19:44:49.957 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:49.959 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.959 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.960 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.974 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:49.974 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:49.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [22ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [24ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [27ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [27ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [42ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [44ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [47ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [48ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1721700812(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1a51810a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76862469 -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328))) after lookup
19:44:49.975 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:49.976 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.977 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:49.977 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:50.095 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:50.095 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:50.096 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=183), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [22ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [24ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [27ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [27ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [42ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [44ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [47ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [48ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [63ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [65ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [65ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [182ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1721700812(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1a51810a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328) -> (continuation of Deferred@1894235819 after retry RPC@176094574), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76862469 -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328) -> (continuation of Deferred@1894235819 after retry RPC@176094574))) after lookup
19:44:50.097 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:50.099 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:50.099 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:50.100 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:50.555 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:50.555 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:44:50.556 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=643), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [22ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [24ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [27ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [27ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [42ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [44ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [47ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [48ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [63ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [65ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [65ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [182ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [184ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [188ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [188ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [637ms] refreshing cache from master, [638ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [642ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1721700812(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1a51810a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328) -> (continuation of Deferred@1894235819 after retry RPC@176094574) -> (continuation of Deferred@108672718 after retry RPC@2073181922), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76862469 -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328) -> (continuation of Deferred@1894235819 after retry RPC@176094574) -> (continuation of Deferred@108672718 after retry RPC@2073181922))) after lookup
19:44:50.558 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:44:50.559 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:50.559 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:50.559 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:44:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:50.880622  8318 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.100706  8344 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fd83ec522d79445cb7c23e85a2486c50)
19:44:51.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.101481  8344 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.104867  8344 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:51.108985  7729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:44:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:51.115383  7729 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:44:51.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.123373  8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a" is_pre_election: true
19:44:51.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.124652  8229 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 6.
19:44:51.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.126363  7729 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a; no voters: fd83ec522d79445cb7c23e85a2486c50
19:44:51.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.127300  8344 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 7 FOLLOWER]: Leader pre-election won for term 8
19:44:51.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.127642  8344 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 7 FOLLOWER]: Starting leader election (detected failure of leader fd83ec522d79445cb7c23e85a2486c50)
19:44:51.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.127990  8344 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 7 FOLLOWER]: Advancing to term 8
19:44:51.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.133545  8344 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:51.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.135227  8344 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 8 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:44:51.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.137066  8229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:44:51.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.137843  8229 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 6 FOLLOWER]: Advancing to term 8
19:44:51.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:51.139302  7729 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:44:51.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.146456  8229 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 8.
19:44:51.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.147696  7729 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a; no voters: fd83ec522d79445cb7c23e85a2486c50
19:44:51.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.148445  8344 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 8 FOLLOWER]: Leader election won for term 8
19:44:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.150540  8344 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 8 LEADER]: Becoming Leader. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Running, Role: LEADER
19:44:51.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.151671  8344 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:51.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.161958  5399 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f reported cstate change: term changed from 7 to 8, leader changed from fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195) to 3cafd375e1de4fac89129b600102f54f (127.4.233.194). New cstate: current_term: 8 leader_uuid: "3cafd375e1de4fac89129b600102f54f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: UNKNOWN } } }
19:44:51.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:51.667062  7729 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f -> Peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Couldn't send request to peer fd83ec522d79445cb7c23e85a2486c50. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:44:51.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.766592  8229 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 8 FOLLOWER]: Refusing update from remote peer 3cafd375e1de4fac89129b600102f54f: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
19:44:51.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:51.768611  8375 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
19:44:52.494 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:52.494 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:52.495 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2582), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [22ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [24ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [27ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [27ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [42ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [44ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [47ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [48ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [61ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [63ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [65ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [65ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [182ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [184ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [188ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [188ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [637ms] refreshing cache from master, [638ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [642ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [645ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [647ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [647ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [2576ms] refreshing cache from master, [2577ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [2581ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1721700812(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@1a51810a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328) -> (continuation of Deferred@1894235819 after retry RPC@176094574) -> (continuation of Deferred@108672718 after retry RPC@2073181922) -> (continuation of Deferred@740826973 after retry RPC@1780535218), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76862469 -> wakeup thread Time-limited test -> (continuation of Deferred@1273090363 after retry RPC@1175578050) -> (continuation of Deferred@1378077035 after retry RPC@466946328) -> (continuation of Deferred@1894235819 after retry RPC@176094574) -> (continuation of Deferred@108672718 after retry RPC@2073181922) -> (continuation of Deferred@740826973 after retry RPC@1780535218))) after lookup
19:44:52.496 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:52.498 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] Successfully connected to peer
19:44:52.520 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xc3cde786, L:/127.0.0.1:35656 - R:/127.4.233.194:40309] using TOKEN/null
19:44:52.574 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.195:34997
19:44:52.574 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:52.576293  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:34997
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45763
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:52.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:52.971626  8418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:52.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:52.972471  8418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:52.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:52.973387  8418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:53.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:53.010600  8418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:53.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:53.011759  8418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.195
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:53.053465  8418 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:34997
19:44:53.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45763
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:53.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:53.055457  8418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:53.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:53.057597  8418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:53.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:53.081334  8435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:54.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:53.081372  8434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:54.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:53.081389  8441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:54.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:54.435266  8436 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1351 milliseconds
19:44:54.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.435452  8418 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:44:54.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.437047  8418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:54.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.440946  8418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:54.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.442471  8418 hybrid_clock.cc:648] HybridClock initialized: now 1786477494442422 us; error 80 us; skew 500 ppm
19:44:54.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.443372  8418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:54.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.451990  8418 webserver.cc:460] Webserver started at http://127.4.233.195:45763/ using document root <none> and password file <none>
19:44:54.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.453131  8418 fs_manager.cc:359] Metadata directory not provided
19:44:54.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.453425  8418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.463653  8418 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.003s
19:44:54.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.469316  8463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:54.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.470675  8418 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
19:44:54.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.471031  8418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:54.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:21 on dist-test-slave-z8x9"
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.473850  8418 fs_report.cc:383] FS layout report
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/data
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:54.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:54.512629  7729 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f -> Peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Couldn't send request to peer fd83ec522d79445cb7c23e85a2486c50. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:44:54.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.544829  8418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:54.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.546947  8418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:54.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.547454  8418 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:54.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.550608  8418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:54.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.557615  8479 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:44:54.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.568899  8418 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:44:54.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.569275  8418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s	user 0.002s	sys 0.001s
19:44:54.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.569536  8418 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:44:54.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.575265  8418 ts_tablet_manager.cc:610] Registered 1 tablets
19:44:54.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.575484  8418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
19:44:54.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.576109  8479 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap starting.
19:44:54.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.649890  8479 log.cc:824] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Log is configured to *not* fsync() on all Append() calls
19:44:54.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.749423  8479 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:44:54.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.750612  8479 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap complete.
19:44:54.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.752887  8479 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent bootstrapping tablet: real 0.177s	user 0.119s	sys 0.051s
19:44:54.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.775175  8479 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:54.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.779665  8479 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd83ec522d79445cb7c23e85a2486c50, State: Initialized, Role: FOLLOWER
19:44:54.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.781409  8479 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:54.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.788266  8479 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent starting tablet: real 0.035s	user 0.024s	sys 0.008s
19:44:54.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.817898  8418 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.195:34997
19:44:54.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.818081  8684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.195:34997 every 8 connection(s)
19:44:54.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.821120  8418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:44:54.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.828706  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 8418
19:44:54.833 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:54.869 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:44:54.870 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.194:40309
19:44:54.870 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:44:54.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.871186  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 7684
19:44:54.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.897220  8686 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:44:54.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.897956  8686 heartbeater.cc:464] Registering TS with master...
19:44:54.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.899691  8686 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:44:54.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.907759  5397 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:54.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.912741  5397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.195:50847
19:44:54.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.916350  8689 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:44:54.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.916749  8689 heartbeater.cc:464] Registering TS with master...
19:44:54.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.918033  8689 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:44:54.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.920624  8690 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:44:54.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.920971  8690 heartbeater.cc:464] Registering TS with master...
19:44:54.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.921803  8690 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:44:54.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.922456  5255 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:54.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:54.925941  5111 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:44:54.938 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] handling channelInactive
19:44:54.938 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state READY due to: connection closed
19:44:54.939 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:44:54.940 [DEBUG - Time-limited test] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:54.942 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.943 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.943 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.955 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:54.955 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:54.955 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1766366893(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@74fdf9d1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74bce67c -> wakeup thread Time-limited test)) after lookup
19:44:54.956 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:54.959 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.959 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.959 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.974 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:54.975 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:54.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1766366893(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@74fdf9d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74bce67c -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861))) after lookup
19:44:54.977 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:54.978 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.978 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:54.978 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.014 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:55.014 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:55.015 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [73ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1766366893(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@74fdf9d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74bce67c -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220))) after lookup
19:44:55.016 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:55.018 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.018 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.018 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.075 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:55.076 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:55.076 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=136), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [73ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [75ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [78ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [129ms] refreshing cache from master, [129ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [134ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1766366893(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@74fdf9d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74bce67c -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110))) after lookup
19:44:55.078 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:55.079 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.079 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.080 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.294 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:55.295 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:55.295 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=355), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [73ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [75ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [78ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [129ms] refreshing cache from master, [129ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [134ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [137ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [139ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [140ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [348ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [354ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1766366893(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@74fdf9d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110) -> (continuation of Deferred@1624762179 after retry RPC@1038571620), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74bce67c -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110) -> (continuation of Deferred@1624762179 after retry RPC@1038571620))) after lookup
19:44:55.297 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:55.299 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.299 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.299 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.854 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:55.855 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:55.855 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=915), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [73ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [75ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [78ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [129ms] refreshing cache from master, [129ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [134ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [137ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [139ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [140ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [348ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [354ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [356ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [359ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [359ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [909ms] refreshing cache from master, [909ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [913ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1766366893(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@74fdf9d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110) -> (continuation of Deferred@1624762179 after retry RPC@1038571620) -> (continuation of Deferred@909969608 after retry RPC@832459283), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74bce67c -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110) -> (continuation of Deferred@1624762179 after retry RPC@1038571620) -> (continuation of Deferred@909969608 after retry RPC@832459283))) after lookup
19:44:55.856 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:44:55.858 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.858 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.858 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:44:55.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:55.922076  8686 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:44:56.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.069535  8718 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:56.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.070036  8718 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:56.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.072691  8718 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:56.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:56.077530  8086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:56.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:56.081525  8086 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:56.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.089138  8594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50" is_pre_election: true
19:44:56.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.090096  8594 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 7.
19:44:56.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.091472  8086 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:44:56.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.092289  8718 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 8 FOLLOWER]: Leader pre-election won for term 9
19:44:56.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.092648  8718 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:44:56.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.092896  8718 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 8 FOLLOWER]: Advancing to term 9
19:44:56.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.098578  8718 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:56.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.100549  8718 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 9 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:44:56.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.101748  8594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:44:56.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.102329  8594 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 7 FOLLOWER]: Advancing to term 9
19:44:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:56.104763  8086 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:44:56.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.111161  8594 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 9.
19:44:56.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.112387  8086 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:44:56.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.113157  8718 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 9 FOLLOWER]: Leader election won for term 9
19:44:56.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.115270  8718 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 9 LEADER]: Becoming Leader. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Running, Role: LEADER
19:44:56.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.116874  8718 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:44:56.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.127480  5397 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a reported cstate change: term changed from 8 to 9, leader changed from 3cafd375e1de4fac89129b600102f54f (127.4.233.194) to e093e47eceec459aa9a5b4c4f354183a (127.4.233.193). New cstate: current_term: 9 leader_uuid: "e093e47eceec459aa9a5b4c4f354183a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: HEALTHY } } }
19:44:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.667620  8594 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 9 FOLLOWER]: Refusing update from remote peer e093e47eceec459aa9a5b4c4f354183a: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
19:44:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:56.669658  8748 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
19:44:56.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:56.731251  8086 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:44:57.974 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:44:57.974 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:44:57.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=3035), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [3ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [14ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [16ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [33ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [73ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [75ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [78ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [129ms] refreshing cache from master, [129ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [134ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [137ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [139ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [140ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [348ms] refreshing cache from master, [349ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [354ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [356ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [359ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [359ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [909ms] refreshing cache from master, [909ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [913ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [916ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [918ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [918ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [3029ms] refreshing cache from master, [3029ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [3033ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1766366893(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@74fdf9d1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110) -> (continuation of Deferred@1624762179 after retry RPC@1038571620) -> (continuation of Deferred@909969608 after retry RPC@832459283) -> (continuation of Deferred@1045835777 after retry RPC@677240137), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74bce67c -> wakeup thread Time-limited test -> (continuation of Deferred@638713117 after retry RPC@898461861) -> (continuation of Deferred@198010192 after retry RPC@1370143220) -> (continuation of Deferred@1562596983 after retry RPC@1497506110) -> (continuation of Deferred@1624762179 after retry RPC@1038571620) -> (continuation of Deferred@909969608 after retry RPC@832459283) -> (continuation of Deferred@1045835777 after retry RPC@677240137))) after lookup
19:44:57.976 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:44:57.978 [DEBUG - kudu-nio-0] (Connection.java:757) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] Successfully connected to peer
19:44:58.008 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xec963cf6, L:/127.0.0.1:54418 - R:/127.4.233.193:33549] using TOKEN/null
19:44:58.073 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.194:40309
19:44:58.073 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:58.076122  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:44:58.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:44:58.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.464154  8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:44:58.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.464774  8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:44:58.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.465457  8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:44:58.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.504029  8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:44:58.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.505153  8794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.194
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:58.548363  8794 tablet_server_runner.cc:78] Tablet server non-default flags:
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:44:58.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:44:58.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:58.550520  8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:44:58.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:58.552695  8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:44:58.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.578433  8810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:59.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:59.256218  8086 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:44:59.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.578776  8808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:59.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:58.578511  8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:44:59.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:44:59.832901  8813 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:44:59.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.833099  8794 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:44:59.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.837431  8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:44:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.840646  8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:44:59.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.842134  8794 hybrid_clock.cc:648] HybridClock initialized: now 1786477499842057 us; error 78 us; skew 500 ppm
19:44:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.842969  8794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:44:59.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.853820  8794 webserver.cc:460] Webserver started at http://127.4.233.194:44553/ using document root <none> and password file <none>
19:44:59.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.855029  8794 fs_manager.cc:359] Metadata directory not provided
19:44:59.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.855324  8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:44:59.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.868149  8794 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.000s	sys 0.007s
19:44:59.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.873725  8847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:44:59.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.875069  8794 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
19:44:59.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.875440  8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:59.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3cafd375e1de4fac89129b600102f54f"
19:44:59.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:18 on dist-test-slave-z8x9"
19:44:59.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.878261  8794 fs_report.cc:383] FS layout report
19:44:59.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:44:59.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:59.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/data
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:44:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:44:59.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.915588  8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:44:59.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.917145  8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:44:59.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.917589  8794 kserver.cc:163] Server-wide thread pool size limit: 3276
19:44:59.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.920286  8794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:44:59.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.926810  8857 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:44:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.938395  8794 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:44:59.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.938699  8794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s	user 0.000s	sys 0.002s
19:44:59.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.939021  8794 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:44:59.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.946695  8794 ts_tablet_manager.cc:610] Registered 1 tablets
19:44:59.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.946962  8794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s	user 0.007s	sys 0.001s
19:44:59.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:44:59.947665  8857 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap starting.
19:45:00.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.026824  8857 log.cc:824] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Log is configured to *not* fsync() on all Append() calls
19:45:00.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.113698  8857 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:45:00.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.114642  8857 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap complete.
19:45:00.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.116102  8857 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent bootstrapping tablet: real 0.169s	user 0.123s	sys 0.036s
19:45:00.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.134891  8857 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:00.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.137244  8857 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Initialized, Role: FOLLOWER
19:45:00.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.138279  8857 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:00.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.142509  8857 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent starting tablet: real 0.026s	user 0.028s	sys 0.000s
19:45:00.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.184201  8794 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.194:40309
19:45:00.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.184398  9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.194:40309 every 8 connection(s)
19:45:00.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.186810  8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:45:00.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.191249  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 8794
19:45:00.192 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:45:00.225 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:45:00.227 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.193:33549
19:45:00.228 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

19:45:00.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.228735  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 8034
19:45:00.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.223574  9055 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:45:00.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.248549  9055 heartbeater.cc:464] Registering TS with master...
19:45:00.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.249835  9055 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:45:00.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.247944  9057 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:45:00.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.252849  9057 heartbeater.cc:464] Registering TS with master...
19:45:00.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.253866  9057 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:45:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.254082  5399 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.258363  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.194:38295
19:45:00.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.258889  5255 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:00.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.260383  9062 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:45:00.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.260748  9062 heartbeater.cc:464] Registering TS with master...
19:45:00.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.261734  9062 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:45:00.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:00.265576  5112 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:00.303 [DEBUG - kudu-nio-0] (Connection.java:239) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] handling channelInactive
19:45:00.304 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state READY due to: connection closed
19:45:00.304 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:45:00.305 [DEBUG - Time-limited test] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:45:00.307 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.307 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.307 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.333 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:00.333 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:00.334 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [2ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [27ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1127911323(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6fcec440 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d741e2 -> wakeup thread Time-limited test)) after lookup
19:45:00.334 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:45:00.336 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.336 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.336 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.352 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:00.352 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:00.352 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [2ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [27ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [29ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [31ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [31ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1127911323(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6fcec440 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d741e2 -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455))) after lookup
19:45:00.353 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:45:00.354 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.354 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.354 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.372 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:00.372 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:00.373 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [2ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [27ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [29ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [31ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [31ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [48ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [49ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [49ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [67ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1127911323(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6fcec440 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d741e2 -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581))) after lookup
19:45:00.373 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:45:00.375 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.375 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.375 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.493 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:00.493 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:00.494 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=189), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [2ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [27ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [29ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [31ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [31ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [48ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [49ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [49ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [67ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [68ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [70ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [70ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [187ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1127911323(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6fcec440 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581) -> (continuation of Deferred@1397713508 after retry RPC@730348165), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d741e2 -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581) -> (continuation of Deferred@1397713508 after retry RPC@730348165))) after lookup
19:45:00.495 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:45:00.496 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.496 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.496 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.813 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:00.813 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:00.814 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=509), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [2ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [27ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [29ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [31ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [31ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [48ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [49ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [49ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [67ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [68ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [70ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [70ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [187ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [189ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [191ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [191ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [504ms] refreshing cache from master, [504ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [507ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1127911323(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6fcec440 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581) -> (continuation of Deferred@1397713508 after retry RPC@730348165) -> (continuation of Deferred@1180869745 after retry RPC@863761920), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d741e2 -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581) -> (continuation of Deferred@1397713508 after retry RPC@730348165) -> (continuation of Deferred@1180869745 after retry RPC@863761920))) after lookup
19:45:00.814 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:45:00.817 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.817 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state CONNECTING due to: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:00.817 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549
19:45:01.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.263943  9055 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:45:01.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.371169  9081 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e093e47eceec459aa9a5b4c4f354183a)
19:45:01.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.371613  9081 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:01.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.373903  9081 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:45:01.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:01.378103  8468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:45:01.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:01.381810  8468 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:45:01.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.387669  8972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "3cafd375e1de4fac89129b600102f54f" is_pre_election: true
19:45:01.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.388543  8972 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd83ec522d79445cb7c23e85a2486c50 in term 8.
19:45:01.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.390036  8468 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50; no voters: e093e47eceec459aa9a5b4c4f354183a
19:45:01.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.390832  9081 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 9 FOLLOWER]: Leader pre-election won for term 10
19:45:01.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.391150  9081 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 9 FOLLOWER]: Starting leader election (detected failure of leader e093e47eceec459aa9a5b4c4f354183a)
19:45:01.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.391566  9081 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 9 FOLLOWER]: Advancing to term 10
19:45:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.396834  9081 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:01.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.398495  9081 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 10 election: Requested vote from peers 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:45:01.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.399439  8972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "fd83ec522d79445cb7c23e85a2486c50" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "3cafd375e1de4fac89129b600102f54f"
19:45:01.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.400080  8972 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 8 FOLLOWER]: Advancing to term 10
19:45:01.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:01.402462  8468 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111)
19:45:01.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.408155  8972 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd83ec522d79445cb7c23e85a2486c50 in term 10.
19:45:01.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.409083  8468 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, fd83ec522d79445cb7c23e85a2486c50; no voters: e093e47eceec459aa9a5b4c4f354183a
19:45:01.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.409869  9081 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 10 FOLLOWER]: Leader election won for term 10
19:45:01.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.411516  9081 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 10 LEADER]: Becoming Leader. State: Replica: fd83ec522d79445cb7c23e85a2486c50, State: Running, Role: LEADER
19:45:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.413043  9081 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:01.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.422883  5399 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 reported cstate change: term changed from 9 to 10, leader changed from e093e47eceec459aa9a5b4c4f354183a (127.4.233.193) to fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195). New cstate: current_term: 10 leader_uuid: "fd83ec522d79445cb7c23e85a2486c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: UNKNOWN } } }
19:45:01.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:01.832587  8468 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 -> Peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Couldn't send request to peer e093e47eceec459aa9a5b4c4f354183a. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:45:01.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.977532  8972 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 10 FOLLOWER]: Refusing update from remote peer fd83ec522d79445cb7c23e85a2486c50: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
19:45:01.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:01.978914  9081 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
19:45:02.873 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:02.873 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:45:02.874 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2569), Traces: [0ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [2ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [27ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [29ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [31ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [31ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [46ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [48ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [49ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [49ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [67ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [68ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [70ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [70ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [187ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [189ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [191ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [191ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [504ms] refreshing cache from master, [504ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [507ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [509ms] sending RPC to server e093e47eceec459aa9a5b4c4f354183a, [512ms] received response from server e093e47eceec459aa9a5b4c4f354183a: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [512ms] delaying RPC due to: Network error: Failed to connect to peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549): Connection refused: /127.4.233.193:33549, [2564ms] refreshing cache from master, [2564ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [2568ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1127911323(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6fcec440 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581) -> (continuation of Deferred@1397713508 after retry RPC@730348165) -> (continuation of Deferred@1180869745 after retry RPC@863761920) -> (continuation of Deferred@946441433 after retry RPC@1422833186), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d741e2 -> wakeup thread Time-limited test -> (continuation of Deferred@776520757 after retry RPC@388581455) -> (continuation of Deferred@500116055 after retry RPC@1481907581) -> (continuation of Deferred@1397713508 after retry RPC@730348165) -> (continuation of Deferred@1180869745 after retry RPC@863761920) -> (continuation of Deferred@946441433 after retry RPC@1422833186))) after lookup
19:45:02.874 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:02.875 [DEBUG - kudu-nio-1] (Connection.java:757) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] Successfully connected to peer
19:45:02.896 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x207f2aa5, L:/127.0.0.1:52610 - R:/127.4.233.195:34997] using TOKEN/null
19:45:02.956 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.193:33549
19:45:02.956 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

19:45:02.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:02.958353  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:45:02.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:45:02.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:45:02.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:45:02.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:45:02.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:45:02.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:33549
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44387
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:45:02.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:45:03.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.302704  9140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:45:03.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.303273  9140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:45:03.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.303936  9140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:45:03.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.335621  9140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:45:03.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.336566  9140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.193
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:03.373128  9140 tablet_server_runner.cc:78] Tablet server non-default flags:
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.193:33549
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.193
19:45:03.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44387
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.193
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/logs
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:45:03.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:03.374410  9140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:45:03.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:03.375996  9140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:45:03.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.396195  9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:04.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:04.232183  8468 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 -> Peer e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549): Couldn't send request to peer e093e47eceec459aa9a5b4c4f354183a. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.193:33549: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:45:04.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.396236  9148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:04.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:03.399302  9140 server_base.cc:1034] running on GCE node
19:45:04.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:03.396924  9147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:04.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.603112  9140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:45:04.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.606253  9140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:45:04.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.607697  9140 hybrid_clock.cc:648] HybridClock initialized: now 1786477504607649 us; error 70 us; skew 500 ppm
19:45:04.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.608513  9140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:45:04.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.615368  9140 webserver.cc:460] Webserver started at http://127.4.233.193:44387/ using document root <none> and password file <none>
19:45:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.616288  9140 fs_manager.cc:359] Metadata directory not provided
19:45:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.616482  9140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:45:04.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.624405  9140 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.002s	sys 0.003s
19:45:04.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.629333  9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:45:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.630532  9140 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.630857  9140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:16 on dist-test-slave-z8x9"
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.633270  9140 fs_report.cc:383] FS layout report
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/wal
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/data
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:45:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:45:04.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:45:04.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:45:04.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:45:04.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:45:04.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:45:04.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.669780  9140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:45:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.671280  9140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:45:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.671695  9140 kserver.cc:163] Server-wide thread pool size limit: 3276
19:45:04.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.674327  9140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:45:04.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.680546  9176 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:45:04.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.688431  9140 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:45:04.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.688656  9140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
19:45:04.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.688913  9140 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:45:04.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.693801  9140 ts_tablet_manager.cc:610] Registered 1 tablets
19:45:04.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.694007  9140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
19:45:04.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.694295  9176 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap starting.
19:45:04.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.769060  9176 log.cc:824] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Log is configured to *not* fsync() on all Append() calls
19:45:04.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.884086  9140 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.193:33549
19:45:04.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.884286  9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.193:33549 every 8 connection(s)
19:45:04.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.886844  9176 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:45:04.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.886842  9140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-0/data/info.pb
19:45:04.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.887692  9176 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Bootstrap complete.
19:45:04.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.889211  9176 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent bootstrapping tablet: real 0.195s	user 0.135s	sys 0.052s
19:45:04.891 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:45:04.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.890266  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 9140
19:45:04.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.918658  9176 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:04.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.922101  9176 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Initialized, Role: FOLLOWER
19:45:04.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.923139  9176 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:04.937 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:45:04.937 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.195:34997
19:45:04.938 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

19:45:04.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.938759  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 8418
19:45:04.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.988420  9176 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a: Time spent starting tablet: real 0.099s	user 0.059s	sys 0.036s
19:45:04.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.991356  9397 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:45:04.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.991915  9397 heartbeater.cc:464] Registering TS with master...
19:45:04.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.993098  9401 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:45:04.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.993327  9397 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:45:04.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.993511  9401 heartbeater.cc:464] Registering TS with master...
19:45:04.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.994410  9401 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:45:04.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.998369  5397 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:45:04.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:04.999245  5256 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:45:05.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:05.005699  5397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.193:39171
19:45:05.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:05.007258  9410 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:45:05.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:05.007623  9410 heartbeater.cc:464] Registering TS with master...
19:45:05.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:05.008656  9410 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:45:05.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:05.012560  5111 ts_manager.cc:194] Re-registered known tserver with Master: e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:45:05.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:05.012970  9397 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:45:05.019 [DEBUG - kudu-nio-1] (Connection.java:239) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] handling channelInactive
19:45:05.019 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state READY due to: connection closed
19:45:05.025 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:45:05.026 [DEBUG - Time-limited test] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:05.028 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.028 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.028 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.033 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:05.033 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:45:05.034 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=8), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@230943818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@261ba89d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5932c351 -> wakeup thread Time-limited test)) after lookup
19:45:05.034 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:05.035 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.035 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.036 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.053 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:05.054 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:45:05.054 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [9ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [11ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [11ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@230943818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@261ba89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5932c351 -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018))) after lookup
19:45:05.055 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:05.058 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.058 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.058 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.093 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:05.093 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:45:05.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [9ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [11ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [11ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [30ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [33ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [33ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@230943818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@261ba89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5932c351 -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799))) after lookup
19:45:05.094 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:05.095 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.096 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.096 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.112 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:05.113 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:45:05.113 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=87), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [9ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [11ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [11ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [30ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [33ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [33ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [69ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [71ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [71ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [87ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@230943818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@261ba89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5932c351 -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661))) after lookup
19:45:05.114 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:05.115 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.115 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.115 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.453 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:05.453 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:45:05.454 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=428), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [9ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [11ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [11ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [30ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [33ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [33ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [69ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [71ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [71ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [87ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [89ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [90ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [90ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [423ms] refreshing cache from master, [424ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [427ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@230943818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@261ba89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661) -> (continuation of Deferred@859136717 after retry RPC@687545188), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5932c351 -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661) -> (continuation of Deferred@859136717 after retry RPC@687545188))) after lookup
19:45:05.454 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:05.455 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.455 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.455 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.653 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:05.654 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)[L]]}]
19:45:05.654 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=628), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [9ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [11ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [11ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [30ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [33ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [33ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [69ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [71ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [71ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [87ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [89ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [90ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [90ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [423ms] refreshing cache from master, [424ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [427ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [429ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [430ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [430ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [624ms] refreshing cache from master, [624ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [628ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@230943818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@261ba89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661) -> (continuation of Deferred@859136717 after retry RPC@687545188) -> (continuation of Deferred@1068411182 after retry RPC@372808960), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5932c351 -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661) -> (continuation of Deferred@859136717 after retry RPC@687545188) -> (continuation of Deferred@1068411182 after retry RPC@372808960))) after lookup
19:45:05.655 [DEBUG - kudu-nio-1] (Connection.java:748) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] connecting to peer
19:45:05.656 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.656 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] cleaning up while in state CONNECTING due to: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:05.656 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997
19:45:06.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.046607  9420 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader fd83ec522d79445cb7c23e85a2486c50)
19:45:06.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.047034  9420 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:06.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.049106  9420 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:45:06.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:06.052366  8854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:45:06.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:06.055856  8854 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:45:06.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.063385  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a" is_pre_election: true
19:45:06.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.064066  9306 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 9.
19:45:06.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.065151  8854 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a; no voters: fd83ec522d79445cb7c23e85a2486c50
19:45:06.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.065902  9420 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 10 FOLLOWER]: Leader pre-election won for term 11
19:45:06.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.066224  9420 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 10 FOLLOWER]: Starting leader election (detected failure of leader fd83ec522d79445cb7c23e85a2486c50)
19:45:06.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.066447  9420 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 10 FOLLOWER]: Advancing to term 11
19:45:06.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.070844  9420 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:06.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.072335  9420 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 11 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), e093e47eceec459aa9a5b4c4f354183a (127.4.233.193:33549)
19:45:06.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.073792  9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "3cafd375e1de4fac89129b600102f54f" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "e093e47eceec459aa9a5b4c4f354183a"
19:45:06.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.074404  9306 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 9 FOLLOWER]: Advancing to term 11
19:45:06.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:06.075281  8854 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111)
19:45:06.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.081131  9306 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cafd375e1de4fac89129b600102f54f in term 11.
19:45:06.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.082101  8854 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cafd375e1de4fac89129b600102f54f, e093e47eceec459aa9a5b4c4f354183a; no voters: fd83ec522d79445cb7c23e85a2486c50
19:45:06.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.082695  9420 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 11 FOLLOWER]: Leader election won for term 11
19:45:06.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.084239  9420 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 11 LEADER]: Becoming Leader. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Running, Role: LEADER
19:45:06.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.085094  9420 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:06.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.094300  5397 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f reported cstate change: term changed from 10 to 11, leader changed from fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195) to 3cafd375e1de4fac89129b600102f54f (127.4.233.194). New cstate: current_term: 11 leader_uuid: "3cafd375e1de4fac89129b600102f54f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: UNKNOWN } } }
19:45:06.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.603188  9306 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 11 FOLLOWER]: Refusing update from remote peer 3cafd375e1de4fac89129b600102f54f: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
19:45:06.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:06.604691  9457 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
19:45:06.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:06.644148  8854 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f -> Peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Couldn't send request to peer fd83ec522d79445cb7c23e85a2486c50. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:45:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.033246  8854 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f -> Peer fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997): Couldn't send request to peer fd83ec522d79445cb7c23e85a2486c50. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.195:34997: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:45:09.052 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:09.053 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:09.053 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=4027), Traces: [0ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [3ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [3ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [8ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [9ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [11ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [11ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [28ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [30ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [33ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [33ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [68ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [69ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [71ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [71ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [87ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [89ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [90ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [90ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [423ms] refreshing cache from master, [424ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [427ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [429ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [430ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [430ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [624ms] refreshing cache from master, [624ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [628ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [629ms] sending RPC to server fd83ec522d79445cb7c23e85a2486c50, [631ms] received response from server fd83ec522d79445cb7c23e85a2486c50: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [631ms] delaying RPC due to: Network error: Failed to connect to peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997): Connection refused: /127.4.233.195:34997, [4023ms] refreshing cache from master, [4024ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [4027ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@230943818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@261ba89d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661) -> (continuation of Deferred@859136717 after retry RPC@687545188) -> (continuation of Deferred@1068411182 after retry RPC@372808960) -> (continuation of Deferred@739477551 after retry RPC@2123485232), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5932c351 -> wakeup thread Time-limited test -> (continuation of Deferred@1874740645 after retry RPC@1562251018) -> (continuation of Deferred@872638114 after retry RPC@424167799) -> (continuation of Deferred@1491803406 after retry RPC@801538661) -> (continuation of Deferred@859136717 after retry RPC@687545188) -> (continuation of Deferred@1068411182 after retry RPC@372808960) -> (continuation of Deferred@739477551 after retry RPC@2123485232))) after lookup
19:45:09.054 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:09.055 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] Successfully connected to peer
19:45:09.077 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x51bd50dc, L:/127.0.0.1:58268 - R:/127.4.233.194:40309] using TOKEN/null
19:45:09.126 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.195:34997
19:45:09.126 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:09.128000  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:34997
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:45:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45763
19:45:09.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:45:09.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:45:09.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:45:09.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:45:09.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:45:09.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.460415  9514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:45:09.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.460992  9514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:45:09.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.461733  9514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:45:09.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.495568  9514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:45:09.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.496891  9514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.195
19:45:09.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:09.534502  9514 tablet_server_runner.cc:78] Tablet server non-default flags:
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.195:34997
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.195
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45763
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.195
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/logs
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:45:09.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:09.535701  9514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:45:09.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:09.537268  9514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:45:09.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.558049  9528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:10.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:09.559860  9514 server_base.cc:1034] running on GCE node
19:45:10.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.559405  9526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:10.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:09.558117  9530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:10.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.805888  9514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:45:10.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.808735  9514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:45:10.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.810190  9514 hybrid_clock.cc:648] HybridClock initialized: now 1786477510810142 us; error 47 us; skew 500 ppm
19:45:10.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.810995  9514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:45:10.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.823421  9514 webserver.cc:460] Webserver started at http://127.4.233.195:45763/ using document root <none> and password file <none>
19:45:10.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.824536  9514 fs_manager.cc:359] Metadata directory not provided
19:45:10.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.824782  9514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:45:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.832816  9514 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
19:45:10.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.837368  9545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:45:10.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.838466  9514 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
19:45:10.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.838871  9514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:45:10.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:45:10.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:21 on dist-test-slave-z8x9"
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.841617  9514 fs_report.cc:383] FS layout report
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/wal
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/data
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:45:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:45:10.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.892727  9514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:45:10.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.894615  9514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:45:10.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.895257  9514 kserver.cc:163] Server-wide thread pool size limit: 3276
19:45:10.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.898792  9514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:45:10.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.906222  9568 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:45:10.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.914492  9514 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:45:10.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.914788  9514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
19:45:10.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.915131  9514 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:45:10.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.920207  9514 ts_tablet_manager.cc:610] Registered 1 tablets
19:45:10.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.920432  9514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
19:45:10.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.920782  9568 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap starting.
19:45:10.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:10.998558  9568 log.cc:824] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Log is configured to *not* fsync() on all Append() calls
19:45:11.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.095723  9568 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:45:11.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.096689  9568 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Bootstrap complete.
19:45:11.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.098685  9568 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent bootstrapping tablet: real 0.178s	user 0.139s	sys 0.035s
19:45:11.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.119506  9568 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:11.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.121901  9568 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd83ec522d79445cb7c23e85a2486c50, State: Initialized, Role: FOLLOWER
19:45:11.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.122896  9568 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.127398  9568 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50: Time spent starting tablet: real 0.028s	user 0.028s	sys 0.000s
19:45:11.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.131891  9514 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.195:34997
19:45:11.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.132095  9779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.195:34997 every 8 connection(s)
19:45:11.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.134501  9514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-2/data/info.pb
19:45:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.142192  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 9514
19:45:11.144 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:45:11.188 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:45:11.193 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.4.233.194:40309
19:45:11.193 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:45:11.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.194267  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 8794
19:45:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.217324  9784 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:45:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.217857  9784 heartbeater.cc:464] Registering TS with master...
19:45:11.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.218417  9782 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:45:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.218792  9782 heartbeater.cc:464] Registering TS with master...
19:45:11.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.219125  9784 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:45:11.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.219856  9782 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:45:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.223941  5397 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:45:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.224385  5256 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:45:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.227429  5397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.195:59403
19:45:11.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.237627  9789 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:45:11.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.238042  9789 heartbeater.cc:464] Registering TS with master...
19:45:11.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.238829  9789 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:45:11.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:11.242986  5111 ts_manager.cc:194] Re-registered known tserver with Master: fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997)
19:45:11.255 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] handling channelInactive
19:45:11.255 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state READY due to: connection closed
19:45:11.257 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:45:11.258 [DEBUG - Time-limited test] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.259 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.259 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.259 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.274 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:11.274 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:11.274 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test)) after lookup
19:45:11.275 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.276 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.276 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.276 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.292 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:11.293 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:11.293 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [18ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [35ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828))) after lookup
19:45:11.294 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.295 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.295 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.295 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.313 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:11.313 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:11.314 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [18ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [35ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021))) after lookup
19:45:11.315 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.315 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.315 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.316 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.393 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:11.394 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:11.394 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=136), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [18ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [35ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [58ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [59ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [136ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071))) after lookup
19:45:11.395 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.396 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.396 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.396 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.533 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:11.533 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:11.534 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=276), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [18ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [35ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [58ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [59ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [136ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [137ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [139ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [139ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [272ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [275ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506))) after lookup
19:45:11.534 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.535 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.535 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.535 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.793 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:11.793 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:11.794 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=536), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [18ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [35ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [58ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [59ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [136ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [137ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [139ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [139ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [272ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [275ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [277ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [278ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [278ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [532ms] refreshing cache from master, [532ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [536ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506) -> (continuation of Deferred@649195371 after retry RPC@235757212), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506) -> (continuation of Deferred@649195371 after retry RPC@235757212))) after lookup
19:45:11.794 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.795 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.795 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.796 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.973 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:11.973 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)[L],e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549),fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:11.974 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=716), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [18ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [35ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [58ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [59ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [136ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [137ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [139ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [139ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [272ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [275ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [277ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [278ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [278ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [532ms] refreshing cache from master, [532ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [536ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [537ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [538ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [539ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [712ms] refreshing cache from master, [712ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [715ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506) -> (continuation of Deferred@649195371 after retry RPC@235757212) -> (continuation of Deferred@137869851 after retry RPC@1814752319), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506) -> (continuation of Deferred@649195371 after retry RPC@235757212) -> (continuation of Deferred@137869851 after retry RPC@1814752319))) after lookup
19:45:11.974 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] connecting to peer
19:45:11.975 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.975 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:11.975 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309) for tablet c09d070664cb49608a8b69571121d26c: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309
19:45:12.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.225265  9811 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:45:12.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.226325  9811 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:12.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.229389  9811 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:12.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.232959  9782 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:45:12.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:12.247426  9171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:45:12.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.253093  9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50" is_pre_election: true
19:45:12.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.254045  9692 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 10.
19:45:12.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:12.255757  9171 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:45:12.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.256628  9171 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:45:12.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.257512  9811 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 11 FOLLOWER]: Leader pre-election won for term 12
19:45:12.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.257833  9811 raft_consensus.cc:482] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 3cafd375e1de4fac89129b600102f54f)
19:45:12.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.258164  9811 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 11 FOLLOWER]: Advancing to term 12
19:45:12.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.265369  9811 raft_consensus.cc:504] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.267376  9811 leader_election.cc:288] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 12 election: Requested vote from peers fd83ec522d79445cb7c23e85a2486c50 (127.4.233.195:34997), 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.268168  9692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09d070664cb49608a8b69571121d26c" candidate_uuid: "e093e47eceec459aa9a5b4c4f354183a" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "fd83ec522d79445cb7c23e85a2486c50"
19:45:12.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.268776  9692 raft_consensus.cc:3037] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 10 FOLLOWER]: Advancing to term 12
19:45:12.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:12.271978  9171 leader_election.cc:334] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111)
19:45:12.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.279460  9692 raft_consensus.cc:2454] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e093e47eceec459aa9a5b4c4f354183a in term 12.
19:45:12.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.280653  9171 leader_election.cc:302] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e093e47eceec459aa9a5b4c4f354183a, fd83ec522d79445cb7c23e85a2486c50; no voters: 3cafd375e1de4fac89129b600102f54f
19:45:12.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.281493  9811 raft_consensus.cc:2781] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 12 FOLLOWER]: Leader election won for term 12
19:45:12.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.283478  9811 raft_consensus.cc:686] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [term 12 LEADER]: Becoming Leader. State: Replica: e093e47eceec459aa9a5b4c4f354183a, State: Running, Role: LEADER
19:45:12.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.285051  9811 consensus_queue.cc:237] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:12.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.293546  5397 catalog_manager.cc:5495] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a reported cstate change: term changed from 11 to 12, leader changed from 3cafd375e1de4fac89129b600102f54f (127.4.233.194) to e093e47eceec459aa9a5b4c4f354183a (127.4.233.193). New cstate: current_term: 12 leader_uuid: "e093e47eceec459aa9a5b4c4f354183a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } health_report { overall_health: HEALTHY } } }
19:45:12.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.730986  9692 raft_consensus.cc:1261] T c09d070664cb49608a8b69571121d26c P fd83ec522d79445cb7c23e85a2486c50 [term 12 FOLLOWER]: Refusing update from remote peer e093e47eceec459aa9a5b4c4f354183a: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
19:45:12.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.732348  9811 consensus_queue.cc:1035] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
19:45:12.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:12.747198  9854 mvcc.cc:205] Tried to move back new op lower bound from 7317411877204865024 to 7317411864931745792. Current Snapshot: MvccSnapshot[applied={T|T < 7317411877204865024}]
19:45:12.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:12.760396  9171 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:45:13.173 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet c09d070664cb49608a8b69571121d26c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373' with partition [<start>, <end>)
19:45:13.173 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c09d070664cb49608a8b69571121d26c@[3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309),e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)[L],fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)]}]
19:45:13.174 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c09d070664cb49608a8b69571121d26c, attempt=16, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1916), Traces: [0ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [2ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [2ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [17ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [18ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [19ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [19ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [35ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [36ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [38ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [56ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [57ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [58ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [59ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [136ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [137ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [139ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [139ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [272ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [275ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [277ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [278ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [278ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [532ms] refreshing cache from master, [532ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [536ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [537ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [538ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [539ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [712ms] refreshing cache from master, [712ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [715ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, [717ms] sending RPC to server 3cafd375e1de4fac89129b600102f54f, [718ms] received response from server 3cafd375e1de4fac89129b600102f54f: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [718ms] delaying RPC due to: Network error: Failed to connect to peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309): Connection refused: /127.4.233.194:40309, [1912ms] refreshing cache from master, [1912ms] Sub RPC GetTableLocations: sending RPC to server master-127.4.233.252:42637, [1915ms] Sub RPC GetTableLocations: received response from server master-127.4.233.252:42637: OK, deferred=Deferred@1548319033(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1985374328@6713b819 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506) -> (continuation of Deferred@649195371 after retry RPC@235757212) -> (continuation of Deferred@137869851 after retry RPC@1814752319) -> (continuation of Deferred@1666380530 after retry RPC@1903248646), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e0ea917 -> wakeup thread Time-limited test -> (continuation of Deferred@702246379 after retry RPC@1428742828) -> (continuation of Deferred@383404064 after retry RPC@83224021) -> (continuation of Deferred@8722345 after retry RPC@2113159071) -> (continuation of Deferred@2062769004 after retry RPC@648518506) -> (continuation of Deferred@649195371 after retry RPC@235757212) -> (continuation of Deferred@137869851 after retry RPC@1814752319) -> (continuation of Deferred@1666380530 after retry RPC@1903248646))) after lookup
19:45:13.175 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] connecting to peer
19:45:13.176 [DEBUG - kudu-nio-0] (Connection.java:757) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] Successfully connected to peer
19:45:13.197 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x5ba17d40, L:/127.0.0.1:52978 - R:/127.4.233.193:33549] using TOKEN/null
19:45:13.242 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.4.233.194:40309
19:45:13.242 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

19:45:13.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:13.244314  5031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:45:13.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu
19:45:13.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:45:13.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:45:13.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:45:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:45:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:45:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:45:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:45:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:45:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:45:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:45:13.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.593513  9872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:45:13.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.594030  9872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:45:13.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.594485  9872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:45:13.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.628067  9872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:45:13.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.629043  9872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.233.194
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:13.667025  9872 tablet_server_runner.cc:78] Tablet server non-default flags:
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.4.233.212:44315
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.4.233.194:40309
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.4.233.194
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44553
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.4.233.254:36095,127.4.233.253:41097,127.4.233.252:42637
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.4.233.194
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/logs
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:45:13.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 18:54:03 UTC on ccycloud.quasar-qqxtnc.root.comops.site
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581135
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:45:13.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:13.668236  9872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:45:13.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:13.669889  9872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:45:13.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.689862  9899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:14.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.689893  9896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:14.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:13.690133  9901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:45:14.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:13.692513  9872 server_base.cc:1034] running on GCE node
19:45:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.909379  9872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:45:14.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.911777  9872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:45:14.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.913285  9872 hybrid_clock.cc:648] HybridClock initialized: now 1786477514913226 us; error 58 us; skew 500 ppm
19:45:14.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.914203  9872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:45:14.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.920758  9872 webserver.cc:460] Webserver started at http://127.4.233.194:44553/ using document root <none> and password file <none>
19:45:14.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.921846  9872 fs_manager.cc:359] Metadata directory not provided
19:45:14.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.922103  9872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:45:14.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.930655  9872 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.008s	sys 0.000s
19:45:14.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.935724  9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
19:45:14.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.936959  9872 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
19:45:14.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.937414  9872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data,/tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:45:14.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3cafd375e1de4fac89129b600102f54f"
19:45:14.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 19:44:18 on dist-test-slave-z8x9"
19:45:14.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.940232  9872 fs_report.cc:383] FS layout report
19:45:14.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/wal
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/data
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:45:14.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:45:14.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.994675  9872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
19:45:14.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.996454  9872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:45:14.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:14.997048  9872 kserver.cc:163] Server-wide thread pool size limit: 3276
19:45:15.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.000463  9872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:45:15.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.008123  9929 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
19:45:15.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.016320  9872 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
19:45:15.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.016613  9872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.000s
19:45:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.017031  9872 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
19:45:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.022933  9872 ts_tablet_manager.cc:610] Registered 1 tablets
19:45:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.023170  9872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
19:45:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.023466  9929 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap starting.
19:45:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.109561  9929 log.cc:824] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Log is configured to *not* fsync() on all Append() calls
19:45:15.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:15.228528  9171 consensus_peers.cc:487] T c09d070664cb49608a8b69571121d26c P e093e47eceec459aa9a5b4c4f354183a -> Peer 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309): Couldn't send request to peer 3cafd375e1de4fac89129b600102f54f. Status: Network error: Client connection negotiation failed: client connection to 127.4.233.194:40309: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
19:45:15.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.284118  9929 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:45:15.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.286091  9929 tablet_bootstrap.cc:492] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Bootstrap complete.
19:45:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.287876  9872 rpc_server.cc:307] RPC server started. Bound to: 127.4.233.194:40309
19:45:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.288108 10156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.233.194:40309 every 8 connection(s)
19:45:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.288609  9929 ts_tablet_manager.cc:1397] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent bootstrapping tablet: real 0.265s	user 0.194s	sys 0.051s
19:45:15.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.291908  9872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE52uFS/test-tmp/mini-kudu-cluster6376437818049954314/ts-1/data/info.pb
19:45:15.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.294950  5031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu as pid 9872
19:45:15.295 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
19:45:15.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.319962  9929 raft_consensus.cc:348] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:15.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.324306  9929 raft_consensus.cc:729] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cafd375e1de4fac89129b600102f54f, State: Initialized, Role: FOLLOWER
19:45:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.325551  9929 consensus_queue.cc:260] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd83ec522d79445cb7c23e85a2486c50" member_type: VOTER last_known_addr { host: "127.4.233.195" port: 34997 } } peers { permanent_uuid: "3cafd375e1de4fac89129b600102f54f" member_type: VOTER last_known_addr { host: "127.4.233.194" port: 40309 } } peers { permanent_uuid: "e093e47eceec459aa9a5b4c4f354183a" member_type: VOTER last_known_addr { host: "127.4.233.193" port: 33549 } }
19:45:15.334 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:45:15.373 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786477462373, tablet=c09d070664cb49608a8b69571121d26c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
19:45:15.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.376753 10158 heartbeater.cc:347] Connected to a master server at 127.4.233.252:42637
19:45:15.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.377563 10158 heartbeater.cc:464] Registering TS with master...
19:45:15.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.379029 10158 heartbeater.cc:511] Master 127.4.233.252:42637 requested a full tablet report, sending...
19:45:15.385 [DEBUG - main] (Connection.java:575) [peer 3cafd375e1de4fac89129b600102f54f(127.4.233.194:40309)] disconnecting while in state TERMINATED
19:45:15.387 [DEBUG - main] (Connection.java:575) [peer master-127.4.233.253:41097(127.4.233.253:41097)] disconnecting while in state READY
19:45:15.388 [DEBUG - main] (Connection.java:575) [peer master-127.4.233.254:36095(127.4.233.254:36095)] disconnecting while in state READY
19:45:15.388 [DEBUG - main] (Connection.java:575) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] disconnecting while in state READY
19:45:15.388 [DEBUG - main] (Connection.java:575) [peer master-127.4.233.252:42637(127.4.233.252:42637)] disconnecting while in state READY
19:45:15.390 [DEBUG - main] (Connection.java:575) [peer fd83ec522d79445cb7c23e85a2486c50(127.4.233.195:34997)] disconnecting while in state TERMINATED
19:45:15.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.384610  5399 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:15.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.393309  9929 ts_tablet_manager.cc:1428] T c09d070664cb49608a8b69571121d26c P 3cafd375e1de4fac89129b600102f54f: Time spent starting tablet: real 0.104s	user 0.054s	sys 0.048s
19:45:15.397 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.4.233.252:42637(127.4.233.252:42637)] handling channelInactive
19:45:15.398 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.4.233.252:42637(127.4.233.252:42637)] cleaning up while in state READY due to: connection closed
19:45:15.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.397847 10160 heartbeater.cc:347] Connected to a master server at 127.4.233.253:41097
19:45:15.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.399045 10160 heartbeater.cc:464] Registering TS with master...
19:45:15.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.399211  5399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.233.194:37313
19:45:15.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.400141 10160 heartbeater.cc:511] Master 127.4.233.253:41097 requested a full tablet report, sending...
19:45:15.401 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
19:45:15.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.404636 10158 heartbeater.cc:503] Master 127.4.233.252:42637 was elected leader, sending a full tablet report...
19:45:15.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.404603  5255 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:15.409 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.4.233.253:41097(127.4.233.253:41097)] handling channelInactive
19:45:15.410 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.4.233.253:41097(127.4.233.253:41097)] cleaning up while in state READY due to: connection closed
19:45:15.410 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.4.233.254:36095(127.4.233.254:36095)] handling channelInactive
19:45:15.410 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.4.233.254:36095(127.4.233.254:36095)] cleaning up while in state READY due to: connection closed
19:45:15.411 [DEBUG - kudu-nio-0] (Connection.java:239) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] handling channelInactive
19:45:15.411 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e093e47eceec459aa9a5b4c4f354183a(127.4.233.193:33549)] cleaning up while in state READY due to: connection closed
19:45:15.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.412305  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 9140
19:45:15.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.430123 10165 heartbeater.cc:347] Connected to a master server at 127.4.233.254:36095
19:45:15.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.430526 10165 heartbeater.cc:464] Registering TS with master...
19:45:15.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.431185 10165 heartbeater.cc:511] Master 127.4.233.254:36095 requested a full tablet report, sending...
19:45:15.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.434333  5112 ts_manager.cc:194] Re-registered known tserver with Master: 3cafd375e1de4fac89129b600102f54f (127.4.233.194:40309)
19:45:15.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.458849  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 9872
19:45:15.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.493005  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 9514
19:45:15.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.536950  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 5055
19:45:15.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.580443  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 5180
19:45:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:15.622699  5376 connection.cc:535] client connection to 127.4.233.253:41097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
19:45:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 19:45:15.623235  5376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
19:45:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 19:45:15.623324  5031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE52uFS/build/tsan/bin/kudu with pid 5319
19:45:16.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T19:45:16Z chronyd exiting

Time: 73.058

OK (1 test)

