JUnit version 4.13.2
.02:12:13.155 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783476733147
02:12:13.217 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
02:12:13.280 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
02:12:13.295 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskttcaKe/test-tmp
02:12:13.302 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
02:12:13.306 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskttcaKe/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
02:12:13.554 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

02:12:13.716 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:12:13.718 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

02:12:13.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.722774 21408 mini_oidc.cc:150] Starting JWKS server
02:12:13.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.726963 21408 webserver.cc:533] Webserver started at http://127.0.0.1:34193/ using document root <none> and password file <none>
02:12:13.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.727497 21408 mini_oidc.cc:186] Starting OIDC Discovery server
02:12:13.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.729806 21408 webserver.cc:533] Webserver started at http://127.0.0.1:41601/ using document root <none> and password file <none>
02:12:13.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:12:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
02:12:13.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:12:13Z Disabled control of system clock
02:12:13.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.788968 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:13.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:13.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/wal
02:12:13.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data
02:12:13.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:13.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:13.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:13.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:13.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:13.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:13.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/logs
02:12:13.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data/info.pb
02:12:13.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:13.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:13.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:13.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:13.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:13.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:13.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:13.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:13.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:12:13.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:13.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:12:13.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:12:13.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.62:43871
02:12:13.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.62
02:12:13.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:13.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:13.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:13.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:13.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:13.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:12:13.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:13.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:12:13.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34193/jwks.json with env {}
02:12:14.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.097826 21420 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:14.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.098416 21420 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:14.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.098927 21420 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:14.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.134318 21420 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:12:14.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.134639 21420 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:14.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.134838 21420 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:12:14.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.135125 21420 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:12:14.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.135391 21420 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:12:14.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.135541 21420 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34193/jwks.json
02:12:14.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.175940 21420 master_runner.cc:387] Master server non-default flags:
02:12:14.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:14.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:14.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:14.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:14.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data
02:12:14.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/wal
02:12:14.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:12:14.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:14.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.62:43871
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34193/jwks.json
02:12:14.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:14.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:14.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data/info.pb
02:12:14.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.62
02:12:14.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/logs
02:12:14.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:14.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:14.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:14.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.177675 21420 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:14.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.179952 21420 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:14.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.193225 21426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.193615 21427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:15.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.197530 21429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.327450 21428 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1130 milliseconds
02:12:15.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.327602 21420 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:15.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.328878 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:15.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.331894 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:15.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.333364 21420 hybrid_clock.cc:648] HybridClock initialized: now 1783476735333286 us; error 100 us; skew 500 ppm
02:12:15.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.334277 21420 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:15.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.341218 21420 webserver.cc:533] Webserver started at http://127.20.232.62:41137/ using document root <none> and password file <none>
02:12:15.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.342327 21420 fs_manager.cc:362] Metadata directory not provided
02:12:15.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.342555 21420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:15.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.342988 21420 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:12:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.347473 21420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data/instance:
02:12:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ac3511b0f9994745bb328d06df231f8b"
02:12:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:15 on dist-test-slave-h94b"
02:12:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.348716 21420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/wal/instance:
02:12:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ac3511b0f9994745bb328d06df231f8b"
02:12:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:15 on dist-test-slave-h94b"
02:12:15.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.356391 21420 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.004s
02:12:15.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.363278 21436 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.364470 21420 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.003s
02:12:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.364774 21420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/wal
02:12:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ac3511b0f9994745bb328d06df231f8b"
02:12:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:15 on dist-test-slave-h94b"
02:12:15.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.365097 21420 fs_report.cc:389] FS layout report
02:12:15.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:15.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/wal
02:12:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/wal
02:12:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data/data
02:12:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:15.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.422993 21420 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.424575 21420 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.425015 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:15.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.498586 21420 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.62:43871
02:12:15.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.498677 21487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.62:43871 every 8 connection(s)
02:12:15.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.501822 21420 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/data/info.pb
02:12:15.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.508735 21488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:12:15.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.510171 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 21420
02:12:15.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.510756 21408 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-0/wal/instance
02:12:15.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.514599 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:15.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:15.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/wal
02:12:15.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data
02:12:15.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:15.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:15.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:15.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:15.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:15.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:15.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/logs
02:12:15.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data/info.pb
02:12:15.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:15.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:15.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:15.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:15.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:15.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:15.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:15.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:15.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:12:15.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:15.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:12:15.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:12:15.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.61:38891
02:12:15.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.61
02:12:15.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:15.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:15.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:15.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:15.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:15.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:12:15.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:15.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:12:15.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34193/jwks.json with env {}
02:12:15.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.522545 21488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } has no permanent_uuid. Determining permanent_uuid...
02:12:15.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.551959 21488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
02:12:15.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.554912 21439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:15.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.560173 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.61:38891: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:15.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.609766 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } attempt: 1
02:12:15.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.614233 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.61:38891: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:15.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.682693 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } attempt: 2
02:12:15.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.687384 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.61:38891: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:15.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.778813 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } attempt: 3
02:12:15.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.783825 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.61:38891: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:15.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.830721 21490 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:15.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.831302 21490 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:15.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.831800 21490 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:15.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.865170 21490 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:12:15.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.865469 21490 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:15.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.865700 21490 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:12:15.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.865898 21490 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:12:15.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.866122 21490 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:12:15.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.866278 21490 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34193/jwks.json
02:12:15.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.904655 21490 master_runner.cc:387] Master server non-default flags:
02:12:15.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:15.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/wal
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:12:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.61:38891
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34193/jwks.json
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data/info.pb
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.61
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:15.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:15.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:15.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:15.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:15.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:15.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/logs
02:12:15.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:12:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:15.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:15.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:15.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:15.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.906052 21490 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:15.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.908082 21490 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:15.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.922468 21499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:15.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.934123 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } attempt: 4
02:12:15.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.941944 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.61:38891: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:16.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.241425 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } attempt: 5
02:12:16.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:16.254966 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.61:38891: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:16.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.802497 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } attempt: 6
02:12:16.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:16.831172 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.61:38891: Network error: Client connection negotiation failed: client connection to 127.20.232.61:38891: connect: Connection refused (error 111)
02:12:17.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.923615 21500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:17.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.276381 21503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:17.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.275619 21501 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1349 milliseconds
02:12:17.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.279543 21490 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.357s	user 0.454s	sys 0.882s
02:12:17.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.281025 21490 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.358s	user 0.455s	sys 0.883s
02:12:17.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.281401 21490 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:17.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.283366 21490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:17.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.286849 21490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:17.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.288832 21490 hybrid_clock.cc:648] HybridClock initialized: now 1783476737288748 us; error 93 us; skew 500 ppm
02:12:17.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.290539 21490 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:17.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.298627 21490 webserver.cc:533] Webserver started at http://127.20.232.61:43019/ using document root <none> and password file <none>
02:12:17.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.299839 21490 fs_manager.cc:362] Metadata directory not provided
02:12:17.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.300088 21490 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:17.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.300733 21490 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:12:17.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.305414 21490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data/instance:
02:12:17.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "007596f7b01b4045b66e66a7675e1013"
02:12:17.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:17 on dist-test-slave-h94b"
02:12:17.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.306599 21490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/wal/instance:
02:12:17.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "007596f7b01b4045b66e66a7675e1013"
02:12:17.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:17 on dist-test-slave-h94b"
02:12:17.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.315224 21490 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.001s	sys 0.005s
02:12:17.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.321415 21511 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:17.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.324169 21490 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.002s	sys 0.001s
02:12:17.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.324572 21490 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/wal
02:12:17.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "007596f7b01b4045b66e66a7675e1013"
02:12:17.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:17 on dist-test-slave-h94b"
02:12:17.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.325038 21490 fs_report.cc:389] FS layout report
02:12:17.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:17.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/wal
02:12:17.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/wal
02:12:17.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data/data
02:12:17.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:17.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:17.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:17.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:17.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:17.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:17.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:17.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:17.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:17.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:17.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:17.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.388085 21490 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.389756 21490 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.390281 21490 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:17.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.462947 21490 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.61:38891
02:12:17.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.463028 21562 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.61:38891 every 8 connection(s)
02:12:17.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.466292 21490 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/data/info.pb
02:12:17.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.473074 21563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:12:17.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.475630 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 21490
02:12:17.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.476207 21408 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-1/wal/instance
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.478603 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/wal
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/logs
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data/info.pb
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.60:33125
02:12:17.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.60
02:12:17.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:17.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:17.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:17.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:17.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:17.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:12:17.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:17.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:12:17.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34193/jwks.json with env {}
02:12:17.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.486745 21563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } has no permanent_uuid. Determining permanent_uuid...
02:12:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.511101 21563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
02:12:17.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.526100 21563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } has no permanent_uuid. Determining permanent_uuid...
02:12:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.528605 21514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:17.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.533237 21563 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.582809 21563 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 1
02:12:17.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.587072 21563 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.655541 21563 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 2
02:12:17.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.659845 21563 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.751358 21563 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 3
02:12:17.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.755798 21563 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:17.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.803411 21565 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:17.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.803953 21565 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:17.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.804411 21565 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:17.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.837697 21565 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:12:17.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.837996 21565 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:17.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.838258 21565 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:12:17.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.838454 21565 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:12:17.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.838634 21565 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:12:17.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.838773 21565 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34193/jwks.json
02:12:17.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.875943 21565 master_runner.cc:387] Master server non-default flags:
02:12:17.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:17.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:17.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:17.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:17.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data
02:12:17.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/wal
02:12:17.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:12:17.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:17.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:17.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:17.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:12:17.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.60:33125
02:12:17.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:12:17.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:17.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:12:17.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34193/jwks.json
02:12:17.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:17.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:17.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data/info.pb
02:12:17.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.60
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/logs
02:12:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:12:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.878674 21565 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:17.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.881222 21565 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:17.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.891667 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } attempt: 7
02:12:17.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.893379 21575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:17.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.899235 21563 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 4
02:12:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.905833 21563 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:17.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.909595 21488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } has no permanent_uuid. Determining permanent_uuid...
02:12:17.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.915874 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:17.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.974349 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 1
02:12:17.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.980545 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:18.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.062120 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 2
02:12:18.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.068162 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:18.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.153708 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 3
02:12:18.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.160164 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:18.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.205338 21563 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 5
02:12:18.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.211134 21563 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:18.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.300711 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 4
02:12:18.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.306571 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:18.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.590116 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 5
02:12:18.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.596071 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:18.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.758672 21563 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 6
02:12:18.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.769203 21563 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:19.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.895815 21576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:19.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.040230 21577 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
02:12:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.044426 21565 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.149s	user 0.004s	sys 0.001s
02:12:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.044880 21565 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.149s	user 0.004s	sys 0.001s
02:12:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.045660 21579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:19.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.045677 21565 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:19.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.047299 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:19.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.050649 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:19.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.052441 21565 hybrid_clock.cc:648] HybridClock initialized: now 1783476739052400 us; error 41 us; skew 500 ppm
02:12:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.053587 21565 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:19.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.063056 21565 webserver.cc:533] Webserver started at http://127.20.232.60:45875/ using document root <none> and password file <none>
02:12:19.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.064479 21565 fs_manager.cc:362] Metadata directory not provided
02:12:19.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.064796 21565 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:19.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.065377 21565 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:12:19.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.071782 21565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data/instance:
02:12:19.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5b201b6193374fbc8f02378e5b49803e"
02:12:19.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:19 on dist-test-slave-h94b"
02:12:19.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.073333 21565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/wal/instance:
02:12:19.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5b201b6193374fbc8f02378e5b49803e"
02:12:19.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:19 on dist-test-slave-h94b"
02:12:19.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.083417 21565 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.007s	sys 0.001s
02:12:19.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.091187 21589 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:19.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.092590 21565 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.003s
02:12:19.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.093007 21565 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/wal
02:12:19.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5b201b6193374fbc8f02378e5b49803e"
02:12:19.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:19 on dist-test-slave-h94b"
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.093472 21565 fs_report.cc:389] FS layout report
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/wal
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/wal
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data/data
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:19.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:19.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.148586 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 6
02:12:19.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.155820 21488 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.20.232.60:33125: Network error: Client connection negotiation failed: client connection to 127.20.232.60:33125: connect: Connection refused (error 111)
02:12:19.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.195582 21565 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:19.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.197271 21565 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:19.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.197688 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:19.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.273932 21565 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.60:33125
02:12:19.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.274057 21642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.60:33125 every 8 connection(s)
02:12:19.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.276893 21565 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/data/info.pb
02:12:19.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.282259 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 21565
02:12:19.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.282752 21408 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/master-2/wal/instance
02:12:19.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.284462 21643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:12:19.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.301012 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } has no permanent_uuid. Determining permanent_uuid...
02:12:19.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.328764 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
02:12:19.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.340938 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } has no permanent_uuid. Determining permanent_uuid...
02:12:19.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.357947 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e: Bootstrap starting.
02:12:19.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.365564 21643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e: Neither blocks nor log segments found. Creating new log.
02:12:19.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.367581 21643 log.cc:826] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e: Log is configured to *not* fsync() on all Append() calls
02:12:19.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.372136 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e: No bootstrap required, opened a new log
02:12:19.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.393700 21643 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:19.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.394425 21643 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:12:19.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.394697 21643 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b201b6193374fbc8f02378e5b49803e, State: Initialized, Role: FOLLOWER
02:12:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.395942 21643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:19.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.399623 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:19.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.400336 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: This master's current role is: FOLLOWER
02:12:19.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.401275 21643 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: configured and running, proceeding with master startup.
02:12:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.425884 21663 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:12:19.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.426303 21663 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:12:19.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.422060 21565 master_runner.cc:187] Error getting master registration for 127.20.232.62:43871: OK, instance_id { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" instance_seqno: 1783476735479688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:12:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.447459 21565 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:12:19.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.452970 21597 tablet.cc:2406] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:12:19.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.532497 21652 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:12:19.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.533357 21652 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:19.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.536738 21652 leader_election.cc:290] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3511b0f9994745bb328d06df231f8b (127.20.232.62:43871), 007596f7b01b4045b66e66a7675e1013 (127.20.232.61:38891)
02:12:19.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.829628 21563 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 7
02:12:19.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.852108 21563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: Bootstrap starting.
02:12:19.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.860862 21563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: Neither blocks nor log segments found. Creating new log.
02:12:19.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.863292 21563 log.cc:826] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: Log is configured to *not* fsync() on all Append() calls
02:12:19.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.871496 21563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: No bootstrap required, opened a new log
02:12:19.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.890841 21563 raft_consensus.cc:359] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:19.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.891695 21563 raft_consensus.cc:385] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:12:19.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.891911 21563 raft_consensus.cc:740] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007596f7b01b4045b66e66a7675e1013, State: Initialized, Role: FOLLOWER
02:12:19.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.892737 21563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:19.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.895469 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:19.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.896201 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [sys.catalog]: This master's current role is: FOLLOWER
02:12:19.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.897539 21563 sys_catalog.cc:565] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [sys.catalog]: configured and running, proceeding with master startup.
02:12:19.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.898350 21517 tablet.cc:2406] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:12:19.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.897734 21538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b201b6193374fbc8f02378e5b49803e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007596f7b01b4045b66e66a7675e1013" is_pre_election: true
02:12:19.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.900030 21538 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b201b6193374fbc8f02378e5b49803e in term 0.
02:12:19.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.901692 21594 leader_election.cc:304] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007596f7b01b4045b66e66a7675e1013, 5b201b6193374fbc8f02378e5b49803e; no voters: 
02:12:19.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.902647 21652 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Leader pre-election won for term 1
02:12:19.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.902984 21652 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:12:19.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.903340 21652 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 0 FOLLOWER]: Advancing to term 1
02:12:19.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.917148 21652 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:19.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.921775 21652 leader_election.cc:290] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [CANDIDATE]: Term 1 election: Requested vote from peers ac3511b0f9994745bb328d06df231f8b (127.20.232.62:43871), 007596f7b01b4045b66e66a7675e1013 (127.20.232.61:38891)
02:12:19.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.923597 21538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b201b6193374fbc8f02378e5b49803e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007596f7b01b4045b66e66a7675e1013"
02:12:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.924201 21538 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [term 0 FOLLOWER]: Advancing to term 1
02:12:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.932042 21538 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b201b6193374fbc8f02378e5b49803e in term 1.
02:12:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.933912 21594 leader_election.cc:304] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007596f7b01b4045b66e66a7675e1013, 5b201b6193374fbc8f02378e5b49803e; no voters: 
02:12:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.934918 21652 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 1 FOLLOWER]: Leader election won for term 1
02:12:19.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.949039 21652 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [term 1 LEADER]: Becoming Leader. State: Replica: 5b201b6193374fbc8f02378e5b49803e, State: Running, Role: LEADER
02:12:19.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.951205 21652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.963101 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b201b6193374fbc8f02378e5b49803e. Latest consensus state: current_term: 1 leader_uuid: "5b201b6193374fbc8f02378e5b49803e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:19.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.963882 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: This master's current role is: LEADER
02:12:19.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.977253 21676 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
02:12:19.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.998332 21681 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:12:20.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.999614 21681 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:12:20.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.004573 21676 catalog_manager.cc:1520] Initializing Kudu cluster ID...
02:12:20.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.985502 21490 master_runner.cc:187] Error getting master registration for 127.20.232.62:43871: OK, instance_id { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" instance_seqno: 1783476735479688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:12:20.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.033806 21490 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:12:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.040066 21538 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [term 1 FOLLOWER]: Refusing update from remote peer 5b201b6193374fbc8f02378e5b49803e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:12:20.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.045181 21652 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [LEADER]: Connected to new peer: Peer: permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:12:20.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.078693 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b201b6193374fbc8f02378e5b49803e. Latest consensus state: current_term: 1 leader_uuid: "5b201b6193374fbc8f02378e5b49803e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:20.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.079457 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [sys.catalog]: This master's current role is: FOLLOWER
02:12:20.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.109908 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b201b6193374fbc8f02378e5b49803e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:20.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.110682 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: This master's current role is: LEADER
02:12:20.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.114405 21676 catalog_manager.cc:1383] Generated new cluster ID: 68d0dcd6509e41088d2138a5f9c45322
02:12:20.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.114728 21676 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
02:12:20.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.118608 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b201b6193374fbc8f02378e5b49803e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:20.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.119318 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013 [sys.catalog]: This master's current role is: FOLLOWER
02:12:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.174284 21676 catalog_manager.cc:1406] Generated new certificate authority record
02:12:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.177830 21676 catalog_manager.cc:1540] Loading token signing keys...
02:12:20.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.189167 21488 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } attempt: 7
02:12:20.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.228786 21676 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e: Generated new TSK 0
02:12:20.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.233554 21676 catalog_manager.cc:1550] Initializing in-progress tserver states...
02:12:20.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.245894 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b: Bootstrap starting.
02:12:20.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.254396 21488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b: Neither blocks nor log segments found. Creating new log.
02:12:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.257458 21488 log.cc:826] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b: Log is configured to *not* fsync() on all Append() calls
02:12:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.270985 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b: No bootstrap required, opened a new log
02:12:20.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.271821 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:20.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:0
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:12:20.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:20.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:20.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:20.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:20.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:20.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:20.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.334437 21488 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:20.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.335325 21488 raft_consensus.cc:385] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:12:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.335620 21488 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3511b0f9994745bb328d06df231f8b, State: Initialized, Role: FOLLOWER
02:12:20.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.342587 21488 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } }
02:12:20.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.367708 21462 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b201b6193374fbc8f02378e5b49803e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3511b0f9994745bb328d06df231f8b"
02:12:20.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.368468 21462 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [term 0 FOLLOWER]: Advancing to term 1
02:12:20.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.374558 21463 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b201b6193374fbc8f02378e5b49803e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3511b0f9994745bb328d06df231f8b" is_pre_election: true
02:12:20.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.377740 21462 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b201b6193374fbc8f02378e5b49803e in term 1.
02:12:20.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.379046 21461 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [term 1 FOLLOWER]: Refusing update from remote peer 5b201b6193374fbc8f02378e5b49803e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:12:20.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.387032 21652 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:12:20.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.366910 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:20.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.389339 21488 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [sys.catalog]: configured and running, proceeding with master startup.
02:12:20.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.389199 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [sys.catalog]: This master's current role is: FOLLOWER
02:12:20.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.457203 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b201b6193374fbc8f02378e5b49803e. Latest consensus state: current_term: 1 leader_uuid: "5b201b6193374fbc8f02378e5b49803e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:20.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.457926 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [sys.catalog]: This master's current role is: FOLLOWER
02:12:20.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.459146 21698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b201b6193374fbc8f02378e5b49803e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:20.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.459863 21698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b [sys.catalog]: This master's current role is: FOLLOWER
02:12:20.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.537043 21707 catalog_manager.cc:1295] Loaded cluster ID: 68d0dcd6509e41088d2138a5f9c45322
02:12:20.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.537838 21707 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b: loading cluster ID for follower catalog manager: success
02:12:20.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.568743 21707 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b: acquiring CA information for follower catalog manager: success
02:12:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.579314 21707 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ac3511b0f9994745bb328d06df231f8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:12:20.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.862535 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b201b6193374fbc8f02378e5b49803e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3511b0f9994745bb328d06df231f8b" member_type: VOTER last_known_addr { host: "127.20.232.62" port: 43871 } } peers { permanent_uuid: "007596f7b01b4045b66e66a7675e1013" member_type: VOTER last_known_addr { host: "127.20.232.61" port: 38891 } } peers { permanent_uuid: "5b201b6193374fbc8f02378e5b49803e" member_type: VOTER last_known_addr { host: "127.20.232.60" port: 33125 } } }
02:12:20.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:20.863835 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b201b6193374fbc8f02378e5b49803e [sys.catalog]: This master's current role is: LEADER
02:12:20.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:20.998180 21690 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:20.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:20.999503 21690 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:21.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.000514 21690 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:21.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.007920 21681 catalog_manager.cc:1295] Loaded cluster ID: 68d0dcd6509e41088d2138a5f9c45322
02:12:21.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.008646 21681 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: loading cluster ID for follower catalog manager: success
02:12:21.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.016476 21681 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: acquiring CA information for follower catalog manager: success
02:12:21.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.035708 21681 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 007596f7b01b4045b66e66a7675e1013: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:12:21.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.063458 21690 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:21.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.065277 21690 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.1
02:12:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.132388 21690 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:0
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:21.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:21.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:21.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:21.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:21.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:21.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:21.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:21.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:21.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.145691 21690 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:21.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.148785 21690 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:21.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.181442 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:22.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.199025 21690 server_base.cc:1061] running on GCE node
02:12:22.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.199613 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:22.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.196763 21720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:22.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.617671 21690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:22.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.620654 21690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:22.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.622534 21690 hybrid_clock.cc:648] HybridClock initialized: now 1783476742622478 us; error 65 us; skew 500 ppm
02:12:22.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.623925 21690 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:22.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.632711 21690 webserver.cc:533] Webserver started at http://127.20.232.1:33627/ using document root <none> and password file <none>
02:12:22.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.634433 21690 fs_manager.cc:362] Metadata directory not provided
02:12:22.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.635053 21690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:22.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.635841 21690 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:12:22.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.645831 21690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/instance:
02:12:22.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "eccfb2228fdd4bc18088d450500f2742"
02:12:22.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:22 on dist-test-slave-h94b"
02:12:22.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.647521 21690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal/instance:
02:12:22.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "eccfb2228fdd4bc18088d450500f2742"
02:12:22.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:22 on dist-test-slave-h94b"
02:12:22.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.658293 21690 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.011s	sys 0.000s
02:12:22.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.666800 21728 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:22.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.668260 21690 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.000s
02:12:22.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.668711 21690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:22.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "eccfb2228fdd4bc18088d450500f2742"
02:12:22.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:22 on dist-test-slave-h94b"
02:12:22.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.669178 21690 fs_report.cc:389] FS layout report
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/data
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:22.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:22.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.748980 21690 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.751119 21690 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:22.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.751726 21690 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:22.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.755358 21690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:22.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.761619 21690 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:12:22.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.762085 21690 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.001s	sys 0.000s
02:12:22.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.762411 21690 ts_tablet_manager.cc:616] Registered 0 tablets
02:12:22.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.762627 21690 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:12:22.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.954105 21690 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.1:38909
02:12:22.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.954172 21840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.1:38909 every 8 connection(s)
02:12:22.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.957893 21690 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:22.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.964089 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 21690
02:12:22.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.964668 21408 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal/instance
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.000653 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:23.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:0
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:23.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:23.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.062834 21843 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:23.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.063097 21842 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:23.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.063673 21842 heartbeater.cc:461] Registering TS with master...
02:12:23.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.065078 21842 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:23.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.066720 21843 heartbeater.cc:461] Registering TS with master...
02:12:23.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.067718 21843 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.070632 21528 ts_manager.cc:194] Registered new tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.071614 21453 ts_manager.cc:194] Registered new tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:23.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.075289 21841 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:23.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.075675 21841 heartbeater.cc:461] Registering TS with master...
02:12:23.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.076572 21841 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:23.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.079320 21608 ts_manager.cc:194] Registered new tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:23.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.081920 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.1:43549
02:12:23.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.374683 21846 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:23.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.375232 21846 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:23.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.375788 21846 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:23.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.410604 21846 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:23.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.411541 21846 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.2
02:12:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.449913 21846 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:12:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:23.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:23.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:23.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:0
02:12:23.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:23.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:23.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:23.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:23.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:23.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:23.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:23.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.451674 21846 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.454107 21846 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:23.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.467983 21857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:24.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:24.085971 21841 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:25.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.470522 21858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:25.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.004371 21859 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1535 milliseconds
02:12:25.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.870784 21856 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 21846
02:12:25.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.003593 21846 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.536s	user 0.552s	sys 0.975s
02:12:25.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.005012 21846 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.538s	user 0.552s	sys 0.975s
02:12:25.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.005671 21846 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:25.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.005736 21860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:25.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.009671 21846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:25.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.011852 21846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:25.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.013388 21846 hybrid_clock.cc:648] HybridClock initialized: now 1783476745013293 us; error 47 us; skew 500 ppm
02:12:25.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.014495 21846 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:25.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.021900 21846 webserver.cc:533] Webserver started at http://127.20.232.2:36149/ using document root <none> and password file <none>
02:12:25.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.023204 21846 fs_manager.cc:362] Metadata directory not provided
02:12:25.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.023447 21846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:25.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.023890 21846 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:12:25.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.028890 21846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/instance:
02:12:25.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:25.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:25 on dist-test-slave-h94b"
02:12:25.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.030053 21846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal/instance:
02:12:25.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:25.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:25 on dist-test-slave-h94b"
02:12:25.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.037752 21846 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
02:12:25.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.045339 21872 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:25.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.046849 21846 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.002s
02:12:25.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.047294 21846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:25.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:25.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:25 on dist-test-slave-h94b"
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.047705 21846 fs_report.cc:389] FS layout report
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/data
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:25.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:25.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.100690 21846 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:25.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.102324 21846 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:25.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.102831 21846 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:25.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.105592 21846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:25.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.110126 21846 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:12:25.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.110358 21846 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:12:25.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.110636 21846 ts_tablet_manager.cc:616] Registered 0 tablets
02:12:25.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.110848 21846 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:12:25.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.249735 21846 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.2:46235
02:12:25.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.249862 21984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.2:46235 every 8 connection(s)
02:12:25.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.252643 21846 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:25.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.263190 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 21846
02:12:25.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.263909 21408 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal/instance
02:12:25.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.281124 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:25.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:25.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:25.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:12:25.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:25.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:25.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:0
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:25.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:25.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:25.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:25.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:25.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:25.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.302598 21987 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:25.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.303632 21987 heartbeater.cc:461] Registering TS with master...
02:12:25.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.307541 21987 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:25.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.310173 21986 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:25.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.310525 21986 heartbeater.cc:461] Registering TS with master...
02:12:25.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.311199 21453 ts_manager.cc:194] Registered new tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:25.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.311394 21986 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:25.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.314412 21528 ts_manager.cc:194] Registered new tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:25.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.315143 21985 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:25.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.315557 21985 heartbeater.cc:461] Registering TS with master...
02:12:25.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.316545 21985 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:25.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.318702 21608 ts_manager.cc:194] Registered new tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:25.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.319989 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.2:39483
02:12:25.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.624730 21995 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:25.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.625227 21995 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:25.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.625636 21995 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:25.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.662166 21995 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:25.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.663115 21995 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.3
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.700551 21995 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:0
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:25.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:25.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.701833 21995 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:25.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:25.703441 21995 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.715781 22001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:26.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.323678 21985 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:27.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:27.119128 22000 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 21995
02:12:27.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:27.213080 21995 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.497s	user 0.647s	sys 0.834s
02:12:27.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:27.213881 21995 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.498s	user 0.647s	sys 0.834s
02:12:27.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.716481 22002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:27.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:27.214929 22003 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1497 milliseconds
02:12:27.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.215808 21995 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:27.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:27.215878 22004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:27.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.219650 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:27.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.221756 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:27.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.223147 21995 hybrid_clock.cc:648] HybridClock initialized: now 1783476747223108 us; error 51 us; skew 500 ppm
02:12:27.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.224023 21995 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:27.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.229851 21995 webserver.cc:533] Webserver started at http://127.20.232.3:37977/ using document root <none> and password file <none>
02:12:27.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.230798 21995 fs_manager.cc:362] Metadata directory not provided
02:12:27.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.231025 21995 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:27.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.231515 21995 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:12:27.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.235762 21995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/instance:
02:12:27.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85d91954824a403e867a4180a90224ea"
02:12:27.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:27 on dist-test-slave-h94b"
02:12:27.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.236935 21995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal/instance:
02:12:27.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85d91954824a403e867a4180a90224ea"
02:12:27.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:27 on dist-test-slave-h94b"
02:12:27.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.243806 21995 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.004s
02:12:27.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.249832 22012 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:27.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.251173 21995 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.251549 21995 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85d91954824a403e867a4180a90224ea"
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:27 on dist-test-slave-h94b"
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.251963 21995 fs_report.cc:389] FS layout report
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/data
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:27.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:27.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:27.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:27.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:27.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:27.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:27.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.309419 21995 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:27.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.311415 21995 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:27.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.312008 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.315023 21995 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:27.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.320549 21995 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:12:27.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.320822 21995 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:12:27.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.321120 21995 ts_tablet_manager.cc:616] Registered 0 tablets
02:12:27.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.321331 21995 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:12:27.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.459336 21995 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.3:34111
02:12:27.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.459425 22124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.3:34111 every 8 connection(s)
02:12:27.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.461966 21995 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:27.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.463737 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 21995
02:12:27.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.464149 21408 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal/instance
02:12:27.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.515797 22125 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:27.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.516340 22125 heartbeater.cc:461] Registering TS with master...
02:12:27.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.517508 22125 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.520419 21608 ts_manager.cc:194] Registered new tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.521652 22127 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.521984 22127 heartbeater.cc:461] Registering TS with master...
02:12:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.522092 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.3:43439
02:12:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.522924 22127 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:27.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.526156 21453 ts_manager.cc:194] Registered new tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:27.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.532681 22126 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:27.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.533244 22126 heartbeater.cc:461] Registering TS with master...
02:12:27.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.534286 22126 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:27.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.537151 21527 ts_manager.cc:194] Registered new tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:27.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.553387 21408 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
02:12:27.564 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:12:27.566 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

02:12:27.576 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.20.232.62"
      port: 43871
    }
    bound_http_address {
      host: "127.20.232.62"
      port: 41137
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.20.232.61"
      port: 38891
    }
    bound_http_address {
      host: "127.20.232.61"
      port: 43019
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.20.232.60"
      port: 33125
    }
    bound_http_address {
      host: "127.20.232.60"
      port: 45875
    }
  }
}

02:12:27.580 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

02:12:27.583 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.20.232.1"
      port: 38909
    }
    bound_http_address {
      host: "127.20.232.1"
      port: 33627
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.20.232.2"
      port: 46235
    }
    bound_http_address {
      host: "127.20.232.2"
      port: 36149
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.20.232.3"
      port: 34111
    }
    bound_http_address {
      host: "127.20.232.3"
      port: 37977
    }
  }
}

02:12:27.583 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
02:12:27.817 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
02:12:27.873 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
02:12:28.053 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.20.232.61:38891(127.20.232.61:38891)] connecting to peer
02:12:28.112 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.20.232.62:43871(127.20.232.62:43871)] connecting to peer
02:12:28.114 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.20.232.60:33125(127.20.232.60:33125)] connecting to peer
02:12:28.133 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.20.232.60:33125(127.20.232.60:33125)] Successfully connected to peer
02:12:28.137 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.20.232.61:38891(127.20.232.61:38891)] Successfully connected to peer
02:12:28.141 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.20.232.62:43871(127.20.232.62:43871)] Successfully connected to peer
02:12:28.242 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.20.232.62
02:12:28.242 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.20.232.60
02:12:28.256 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.20.232.61
02:12:28.490 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0xd9cb178d, L:/127.0.0.1:60186 - R:/127.20.232.60:33125] using SASL/PLAIN
02:12:28.490 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xbb0916ac, L:/127.0.0.1:36658 - R:/127.20.232.62:43871] using SASL/PLAIN
02:12:28.490 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x5b51b52a, L:/127.0.0.1:38672 - R:/127.20.232.61:38891] using SASL/PLAIN
02:12:28.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.526809 22125 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:28.542 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
02:12:28.543 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=530), Traces: [0ms] refreshing cache from master, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.20.232.61:38891, [96ms] Sub RPC ConnectToMaster: sending RPC to server master-127.20.232.62:43871, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.20.232.60:33125, [491ms] Sub RPC ConnectToMaster: received response from server master-127.20.232.61:38891: OK, [491ms] Sub RPC ConnectToMaster: received response from server master-127.20.232.60:33125: OK, [495ms] Sub RPC ConnectToMaster: received response from server master-127.20.232.62:43871: OK, deferred=null) after lookup
02:12:28.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.595695 21608 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60186:
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998"
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:12:28.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:12:28.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
02:12:28.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
02:12:28.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
02:12:28.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
02:12:28.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:12:28.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:12:28.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:28.600137 21608 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
02:12:28.685 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table d4aa2562576a4b01bb915bbaf12d4628
02:12:28.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.716048 22060 tablet_service.cc:1511] Processing CreateTablet for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998 [id=d4aa2562576a4b01bb915bbaf12d4628]), partition=RANGE (key) PARTITION UNBOUNDED
02:12:28.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.719111 22060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73. 1 dirs total, 0 dirs full, 0 dirs failed
02:12:28.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.730638 21920 tablet_service.cc:1511] Processing CreateTablet for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998 [id=d4aa2562576a4b01bb915bbaf12d4628]), partition=RANGE (key) PARTITION UNBOUNDED
02:12:28.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.731140 21776 tablet_service.cc:1511] Processing CreateTablet for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998 [id=d4aa2562576a4b01bb915bbaf12d4628]), partition=RANGE (key) PARTITION UNBOUNDED
02:12:28.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.732285 21920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73. 1 dirs total, 0 dirs full, 0 dirs failed
02:12:28.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.732956 21776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73. 1 dirs total, 0 dirs full, 0 dirs failed
02:12:28.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.750517 22160 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap starting.
02:12:28.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.760586 22161 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap starting.
02:12:28.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.764945 22160 tablet_bootstrap.cc:654] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Neither blocks nor log segments found. Creating new log.
02:12:28.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.767807 22160 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Log is configured to *not* fsync() on all Append() calls
02:12:28.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.768712 22161 tablet_bootstrap.cc:654] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Neither blocks nor log segments found. Creating new log.
02:12:28.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.769351 22162 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap starting.
02:12:28.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.771489 22161 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Log is configured to *not* fsync() on all Append() calls
02:12:28.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.774412 22160 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: No bootstrap required, opened a new log
02:12:28.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.775192 22160 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent bootstrapping tablet: real 0.026s	user 0.016s	sys 0.000s
02:12:28.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.779963 22161 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: No bootstrap required, opened a new log
02:12:28.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.780766 22161 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.000s
02:12:28.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.782670 22162 tablet_bootstrap.cc:654] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Neither blocks nor log segments found. Creating new log.
02:12:28.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.785009 22162 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Log is configured to *not* fsync() on all Append() calls
02:12:28.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.790727 22162 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: No bootstrap required, opened a new log
02:12:28.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.791281 22162 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.003s
02:12:28.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.802896 22160 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.803856 22160 raft_consensus.cc:385] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:12:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.804144 22160 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d91954824a403e867a4180a90224ea, State: Initialized, Role: FOLLOWER
02:12:28.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.803494 22161 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:28.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.805404 22161 raft_consensus.cc:385] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:12:28.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.805075 22160 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:28.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.805984 22161 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Initialized, Role: FOLLOWER
02:12:28.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.807394 22161 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:28.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.809049 22160 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent starting tablet: real 0.033s	user 0.033s	sys 0.000s
02:12:28.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.820619 22162 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:28.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.821563 22162 raft_consensus.cc:385] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:12:28.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.821877 22162 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Initialized, Role: FOLLOWER
02:12:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.823215 22162 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.825138 22161 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent starting tablet: real 0.044s	user 0.030s	sys 0.004s
02:12:28.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.839171 22162 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent starting tablet: real 0.048s	user 0.026s	sys 0.011s
02:12:28.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.983541 22168 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:12:28.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.984233 22168 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:28.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.986980 22168 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:28.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:28.988310 22132 tablet.cc:2406] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:12:28.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.997670 22080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea" is_pre_election: true
02:12:28.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.998621 22080 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 0.
02:12:29.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.000061 21732 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742; no voters: 
02:12:29.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.000231 21940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" is_pre_election: true
02:12:29.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.001014 22168 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:12:29.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.001053 21940 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 0.
02:12:29.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.001309 22168 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:12:29.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.001655 22168 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 0 FOLLOWER]: Advancing to term 1
02:12:29.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.006367 22168 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:29.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.007921 22168 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 1 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:29.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.009001 22080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea"
02:12:29.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.009325 21940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:29.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.009481 22080 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 0 FOLLOWER]: Advancing to term 1
02:12:29.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.009737 21940 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 0 FOLLOWER]: Advancing to term 1
02:12:29.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.014474 21940 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 1.
02:12:29.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.014688 22080 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 1.
02:12:29.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.015585 21732 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742; no voters: 
02:12:29.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.016289 22168 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 1 FOLLOWER]: Leader election won for term 1
02:12:29.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.018110 22168 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 1 LEADER]: Becoming Leader. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Running, Role: LEADER
02:12:29.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.019153 22168 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:29.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.021417 21990 tablet.cc:2406] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:12:29.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.032960 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 reported cstate change: term changed from 0 to 1, leader changed from <none> to eccfb2228fdd4bc18088d450500f2742 (127.20.232.1). New cstate: current_term: 1 leader_uuid: "eccfb2228fdd4bc18088d450500f2742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: UNKNOWN } } }
02:12:29.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.073989 21849 tablet.cc:2406] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:12:29.163 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table d4aa2562576a4b01bb915bbaf12d4628
02:12:29.183 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:29.191 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299992, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:29.192 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [10ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=null) after lookup
02:12:29.205 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:29.208 [DEBUG - kudu-nio-3] (Connection.java:757) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] Successfully connected to peer
02:12:29.288 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x5641636e, L:/127.0.0.1:57578 - R:/127.20.232.1:38909] using TOKEN/null
02:12:29.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.315194 21940 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 1 FOLLOWER]: Refusing update from remote peer eccfb2228fdd4bc18088d450500f2742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:12:29.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.317891 22080 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 1 FOLLOWER]: Refusing update from remote peer eccfb2228fdd4bc18088d450500f2742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:12:29.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.317219 22168 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
02:12:29.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.323339 22171 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
02:12:29.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.406278 22180 mvcc.cc:204] Tried to move back new op lower bound from 7305120765170950144 to 7305120763990958080. Current Snapshot: MvccSnapshot[applied={T|T < 7305120765170950144}]
02:12:29.588 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:29.590 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.1:38909
02:12:29.591 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:29.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.592201 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 21690
02:12:29.616 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:29.618 [DEBUG - kudu-nio-3] (Connection.java:239) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] handling channelInactive
02:12:29.619 [DEBUG - kudu-nio-3] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state READY due to: connection closed
02:12:29.619 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: connection closed
02:12:29.640 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:29.641 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:29.641 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [23ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030076872(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5d869e7f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60f4632f -> wakeup thread Time-limited test)) after lookup
02:12:29.642 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:29.646 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.646 [DEBUG - kudu-nio-4] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.646 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.660 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:29.660 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:29.661 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [23ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [29ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [42ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030076872(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5d869e7f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60f4632f -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065))) after lookup
02:12:29.662 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:29.664 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.664 [DEBUG - kudu-nio-5] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.665 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:29.698 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:29.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=82), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [23ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [29ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [42ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [44ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [48ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [48ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [76ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [80ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030076872(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5d869e7f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60f4632f -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553))) after lookup
02:12:29.700 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:29.702 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.702 [DEBUG - kudu-nio-6] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.703 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.819 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:29.820 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:29.820 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=203), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [23ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [29ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [42ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [44ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [48ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [48ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [76ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [80ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [83ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [86ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [86ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [197ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [202ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030076872(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5d869e7f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553) -> (continuation of Deferred@2019004677 after retry RPC@589782209), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60f4632f -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553) -> (continuation of Deferred@2019004677 after retry RPC@589782209))) after lookup
02:12:29.821 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:29.823 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.824 [DEBUG - kudu-nio-7] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:29.824 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:30.299 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:30.299 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:30.300 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=682), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [23ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [29ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [42ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [44ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [48ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [48ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [76ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [80ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [83ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [86ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [86ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [197ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [202ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [204ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [207ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [207ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [677ms] refreshing cache from master, [677ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [681ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030076872(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5d869e7f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553) -> (continuation of Deferred@2019004677 after retry RPC@589782209) -> (continuation of Deferred@398870255 after retry RPC@93560778), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60f4632f -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553) -> (continuation of Deferred@2019004677 after retry RPC@589782209) -> (continuation of Deferred@398870255 after retry RPC@93560778))) after lookup
02:12:30.301 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:30.303 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:30.303 [DEBUG - kudu-nio-0] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:30.303 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:30.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.970621 22193 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:12:30.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.971072 22193 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:30.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.973511 22193 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:30.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.975207 22194 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:12:30.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.976049 22194 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:30.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:30.978256 21876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:30.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.979473 22194 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:30.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:30.984308 22016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:30.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:30.988755 21876 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:30.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.990746 22080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea" is_pre_election: true
02:12:30.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:30.991312 22016 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:30.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.991822 22080 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 1.
02:12:30.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.993008 21876 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:12:30.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.993803 22193 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 1 FOLLOWER]: Leader pre-election won for term 2
02:12:30.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.994138 22193 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:12:30.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.994479 22193 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 1 FOLLOWER]: Advancing to term 2
02:12:30.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.998454 21940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "85d91954824a403e867a4180a90224ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" is_pre_election: true
02:12:31.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:30.999773 22193 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:31.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.000818 21940 raft_consensus.cc:2393] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85d91954824a403e867a4180a90224ea in current term 2: Already voted for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in this term.
02:12:31.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.001456 22193 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 2 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:31.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.002367 22080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea"
02:12:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.002934 22080 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 1 FOLLOWER]: Advancing to term 2
02:12:31.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.004501 22016 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85d91954824a403e867a4180a90224ea; no voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742
02:12:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:31.005903 21876 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:31.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.008939 22080 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 2.
02:12:31.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.009531 22194 raft_consensus.cc:2749] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
02:12:31.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.009977 21876 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:12:31.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.010692 22193 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Leader election won for term 2
02:12:31.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.012625 22193 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 LEADER]: Becoming Leader. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Running, Role: LEADER
02:12:31.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.013562 22193 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:31.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.022403 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 reported cstate change: term changed from 1 to 2, leader changed from eccfb2228fdd4bc18088d450500f2742 (127.20.232.1) to 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2). New cstate: current_term: 2 leader_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: HEALTHY } } }
02:12:31.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.489331 22080 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Refusing update from remote peer 75d6e7b68fdf468cb97ac9fa184bec55: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
02:12:31.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.490934 22193 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
02:12:31.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:31.529240 21876 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 -> Peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Couldn't send request to peer eccfb2228fdd4bc18088d450500f2742. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:31.999 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:31.999 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:32.000 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2382), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [23ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [29ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [42ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [44ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [48ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [48ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [76ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [80ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [83ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [86ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [86ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [197ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [202ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [204ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [207ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [207ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [677ms] refreshing cache from master, [677ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [681ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [684ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [686ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [686ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2376ms] refreshing cache from master, [2377ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [2381ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030076872(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5d869e7f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553) -> (continuation of Deferred@2019004677 after retry RPC@589782209) -> (continuation of Deferred@398870255 after retry RPC@93560778) -> (continuation of Deferred@1069294788 after retry RPC@642285878), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60f4632f -> wakeup thread Time-limited test -> (continuation of Deferred@621167907 after retry RPC@1819012065) -> (continuation of Deferred@1104558241 after retry RPC@1612415553) -> (continuation of Deferred@2019004677 after retry RPC@589782209) -> (continuation of Deferred@398870255 after retry RPC@93560778) -> (continuation of Deferred@1069294788 after retry RPC@642285878))) after lookup
02:12:32.001 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:32.002 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] Successfully connected to peer
02:12:32.033 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x1c0d679b, L:/127.0.0.1:47966 - R:/127.20.232.2:46235] using TOKEN/null
02:12:32.085 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.1:38909
02:12:32.086 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.087904 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:12:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:32.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:12:32.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:12:32.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:32.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:32.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:32.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:32.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.411386 22229 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:32.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.411935 22229 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:32.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.412566 22229 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:32.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.446554 22229 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:32.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.447532 22229 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.1
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.488749 22229 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:32.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:12:32.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.490523 22229 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:32.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.492520 22229 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:32.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.513830 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:34.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:34.062984 21876 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 -> Peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Couldn't send request to peer eccfb2228fdd4bc18088d450500f2742. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.916208 22234 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 22229
02:12:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:34.101428 22229 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.586s	user 0.575s	sys 0.998s
02:12:34.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:34.101806 22229 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.586s	user 0.575s	sys 0.998s
02:12:34.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.516986 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:34.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:34.104169 22238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:34.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:34.107736 22237 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1587 milliseconds
02:12:34.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.108647 22229 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:34.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.110188 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:34.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.112756 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:34.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.114415 22229 hybrid_clock.cc:648] HybridClock initialized: now 1783476754114375 us; error 40 us; skew 500 ppm
02:12:34.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.115496 22229 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:34.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.121716 22229 webserver.cc:533] Webserver started at http://127.20.232.1:33627/ using document root <none> and password file <none>
02:12:34.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.122634 22229 fs_manager.cc:362] Metadata directory not provided
02:12:34.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.122854 22229 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:34.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.130321 22229 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
02:12:34.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.134897 22254 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:34.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.136593 22229 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
02:12:34.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.137038 22229 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:34.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "eccfb2228fdd4bc18088d450500f2742"
02:12:34.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:22 on dist-test-slave-h94b"
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.139314 22229 fs_report.cc:389] FS layout report
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/data
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:34.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:34.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:34.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:34.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:34.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:34.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.194455 22229 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:34.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.196017 22229 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:34.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.196456 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:34.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.199498 22229 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:34.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.205298 22262 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:34.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.213719 22229 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:34.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.214004 22229 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
02:12:34.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.214350 22229 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:34.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.219381 22229 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:34.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.219614 22229 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.003s	sys 0.001s
02:12:34.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.220077 22262 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap starting.
02:12:34.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.283205 22262 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Log is configured to *not* fsync() on all Append() calls
02:12:34.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.320361 22262 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:34.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.321842 22262 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap complete.
02:12:34.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.323856 22262 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent bootstrapping tablet: real 0.104s	user 0.059s	sys 0.035s
02:12:34.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.346294 22262 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:34.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.348681 22262 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Initialized, Role: FOLLOWER
02:12:34.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.349907 22262 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:34.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.354398 22262 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent starting tablet: real 0.030s	user 0.025s	sys 0.004s
02:12:34.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.392112 22229 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.1:38909
02:12:34.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.392338 22370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.1:38909 every 8 connection(s)
02:12:34.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.394824 22229 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:34.403 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:34.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.402078 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 22229
02:12:34.428 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:34.429 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.2:46235
02:12:34.429 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.430915 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 21846
02:12:34.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.447204 22372 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:34.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.447373 22371 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:34.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.447770 22371 heartbeater.cc:461] Registering TS with master...
02:12:34.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.447767 22372 heartbeater.cc:461] Registering TS with master...
02:12:34.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.449285 22371 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:34.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.449894 22372 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:34.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.451499 22373 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:34.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.451865 22373 heartbeater.cc:461] Registering TS with master...
02:12:34.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.452829 22373 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:34.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.455550 21608 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:34.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.456238 21528 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:34.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.457140 21452 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:34.461 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] handling channelInactive
02:12:34.462 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state READY due to: connection closed
02:12:34.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:34.461946 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.1:36397
02:12:34.467 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:34.468 [DEBUG - Time-limited test] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:34.469 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.469 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.470 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.480 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:34.480 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:34.481 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [3ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030353359(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@1fcdc846 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38e7126 -> wakeup thread Time-limited test)) after lookup
02:12:34.481 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:34.483 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.483 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.483 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:34.500 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:34.500 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [3ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [14ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [16ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [16ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030353359(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@1fcdc846 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38e7126 -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583))) after lookup
02:12:34.502 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:34.504 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.504 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.504 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.519 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:34.520 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:34.520 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [3ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [14ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [16ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [16ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [37ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [37ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [52ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030353359(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@1fcdc846 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38e7126 -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086))) after lookup
02:12:34.521 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:34.523 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.523 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.523 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.639 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:34.640 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:34.640 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=173), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [3ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [14ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [16ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [16ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [37ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [37ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [52ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [54ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [56ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [56ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [171ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030353359(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@1fcdc846 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38e7126 -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822))) after lookup
02:12:34.642 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:34.643 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.643 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:34.643 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:35.160 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:35.160 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:35.161 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=693), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [3ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [14ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [16ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [16ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [37ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [37ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [52ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [54ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [56ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [56ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [171ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [174ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [176ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [176ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [687ms] refreshing cache from master, [688ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [693ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030353359(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@1fcdc846 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822) -> (continuation of Deferred@74922015 after retry RPC@1324586655), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38e7126 -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822) -> (continuation of Deferred@74922015 after retry RPC@1324586655))) after lookup
02:12:35.162 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:35.163 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:35.164 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:35.164 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:35.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.466195 22371 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:35.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:35.578 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:35.579 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1111), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [3ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [14ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [16ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [16ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [37ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [37ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [52ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [54ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [56ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [56ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [171ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [174ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [176ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [176ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [687ms] refreshing cache from master, [688ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [693ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [694ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [697ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [697ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [1106ms] refreshing cache from master, [1107ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [1110ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030353359(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@1fcdc846 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822) -> (continuation of Deferred@74922015 after retry RPC@1324586655) -> (continuation of Deferred@465202023 after retry RPC@625022427), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38e7126 -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822) -> (continuation of Deferred@74922015 after retry RPC@1324586655) -> (continuation of Deferred@465202023 after retry RPC@625022427))) after lookup
02:12:35.579 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:35.580 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:35.580 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:35.581 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.673980 22383 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 75d6e7b68fdf468cb97ac9fa184bec55)
02:12:35.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.674760 22383 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:35.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.676842 22383 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:35.686455 22016 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:12:35.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.691542 22325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "85d91954824a403e867a4180a90224ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "eccfb2228fdd4bc18088d450500f2742" is_pre_election: true
02:12:35.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.692364 22325 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d91954824a403e867a4180a90224ea in term 1.
02:12:35.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.693697 22016 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742; no voters: 75d6e7b68fdf468cb97ac9fa184bec55
02:12:35.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.694440 22383 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Leader pre-election won for term 3
02:12:35.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.694810 22383 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Starting leader election (detected failure of leader 75d6e7b68fdf468cb97ac9fa184bec55)
02:12:35.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.695158 22383 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 2 FOLLOWER]: Advancing to term 3
02:12:35.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.700192 22383 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:35.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.701798 22383 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 3 election: Requested vote from peers eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:35.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.702662 22325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "85d91954824a403e867a4180a90224ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "eccfb2228fdd4bc18088d450500f2742"
02:12:35.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.703230 22325 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 1 FOLLOWER]: Advancing to term 3
02:12:35.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:35.706243 22016 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:12:35.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.709998 22325 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d91954824a403e867a4180a90224ea in term 3.
02:12:35.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.711125 22016 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742; no voters: 75d6e7b68fdf468cb97ac9fa184bec55
02:12:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.711773 22383 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 3 FOLLOWER]: Leader election won for term 3
02:12:35.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.712930 22383 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 3 LEADER]: Becoming Leader. State: Replica: 85d91954824a403e867a4180a90224ea, State: Running, Role: LEADER
02:12:35.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.714318 22383 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:35.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.724807 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea reported cstate change: term changed from 2 to 3, leader changed from 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2) to 85d91954824a403e867a4180a90224ea (127.20.232.3). New cstate: current_term: 3 leader_uuid: "85d91954824a403e867a4180a90224ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: UNKNOWN } } }
02:12:36.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.151516 22325 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 3 FOLLOWER]: Refusing update from remote peer 85d91954824a403e867a4180a90224ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
02:12:36.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.153541 22383 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
02:12:36.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:36.181926 22016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111) [suppressed 2 similar messages]
02:12:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:36.187736 22016 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea -> Peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Couldn't send request to peer 75d6e7b68fdf468cb97ac9fa184bec55. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:37.981 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:37.981 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:37.981 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=3514), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [3ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [12ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [14ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [16ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [16ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [31ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [37ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [37ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [52ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [54ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [56ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [56ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [167ms] refreshing cache from master, [167ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [171ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [174ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [176ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [176ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [687ms] refreshing cache from master, [688ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [693ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [694ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [697ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [697ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [1106ms] refreshing cache from master, [1107ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [1110ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [1112ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [1114ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [1114ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [3507ms] refreshing cache from master, [3507ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [3513ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030353359(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@1fcdc846 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822) -> (continuation of Deferred@74922015 after retry RPC@1324586655) -> (continuation of Deferred@465202023 after retry RPC@625022427) -> (continuation of Deferred@385824247 after retry RPC@1002243390), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38e7126 -> wakeup thread Time-limited test -> (continuation of Deferred@1265871438 after retry RPC@990176583) -> (continuation of Deferred@28825234 after retry RPC@495265086) -> (continuation of Deferred@383085407 after retry RPC@575475822) -> (continuation of Deferred@74922015 after retry RPC@1324586655) -> (continuation of Deferred@465202023 after retry RPC@625022427) -> (continuation of Deferred@385824247 after retry RPC@1002243390))) after lookup
02:12:37.982 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:37.984 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] Successfully connected to peer
02:12:38.014 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x02c70e9d, L:/127.0.0.1:40680 - R:/127.20.232.3:34111] using TOKEN/null
02:12:38.073 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.2:46235
02:12:38.074 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:38.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:38.075419 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:38.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:38.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:38.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:46235
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36149
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:38.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.408212 22426 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:38.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.408743 22426 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:38.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.409356 22426 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:38.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.442548 22426 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:38.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.443487 22426 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.2
02:12:38.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:38.491879 22426 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:38.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:38.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:38.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:38.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:38.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:12:38.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:46235
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36149
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:38.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:38.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:38.493770 22426 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:38.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:38.496023 22426 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:38.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.512043 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:38.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.698359 22016 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea -> Peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Couldn't send request to peer 75d6e7b68fdf468cb97ac9fa184bec55. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:39.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:38.515009 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:39.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.686558 22426 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.173s	user 0.483s	sys 0.688s
02:12:39.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.686959 22426 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.174s	user 0.483s	sys 0.688s
02:12:39.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.688411 22426 server_base.cc:1061] running on GCE node
02:12:39.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.690310 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:39.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.691752 22426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:39.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.694167 22426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:39.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.697701 22426 hybrid_clock.cc:648] HybridClock initialized: now 1783476759697625 us; error 112 us; skew 500 ppm
02:12:39.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.698807 22426 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:39.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.706418 22426 webserver.cc:533] Webserver started at http://127.20.232.2:36149/ using document root <none> and password file <none>
02:12:39.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.707496 22426 fs_manager.cc:362] Metadata directory not provided
02:12:39.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.707752 22426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:39.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.717607 22426 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.001s
02:12:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.722595 22444 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:39.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.724150 22426 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.004s
02:12:39.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.724578 22426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:39.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:39.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:25 on dist-test-slave-h94b"
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.727480 22426 fs_report.cc:389] FS layout report
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/data
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:39.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:39.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.810895 22426 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:39.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.812424 22426 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:39.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.812835 22426 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:39.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.815554 22426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:39.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.821314 22451 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:39.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.830192 22426 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:39.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.830454 22426 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
02:12:39.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.830680 22426 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:39.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.835590 22426 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:39.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.835804 22426 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
02:12:39.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.836333 22451 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap starting.
02:12:39.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.888718 22451 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Log is configured to *not* fsync() on all Append() calls
02:12:39.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.925199 22451 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:39.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.926096 22451 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap complete.
02:12:39.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.927398 22451 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent bootstrapping tablet: real 0.092s	user 0.077s	sys 0.012s
02:12:39.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.946352 22451 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:39.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.949147 22451 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Initialized, Role: FOLLOWER
02:12:39.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.950248 22451 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:39.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.954617 22451 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent starting tablet: real 0.027s	user 0.026s	sys 0.001s
02:12:40.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.002970 22426 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.2:46235
02:12:40.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.003342 22559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.2:46235 every 8 connection(s)
02:12:40.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.005672 22426 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:40.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.014297 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 22426
02:12:40.015 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:40.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.050195 22560 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:40.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.050688 22560 heartbeater.cc:461] Registering TS with master...
02:12:40.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.052037 22560 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:40.052 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:40.053 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.3:34111
02:12:40.053 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:40.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.054205 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 21995
02:12:40.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.057561 21606 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.060545 21606 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.2:55663
02:12:40.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.063820 22561 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:40.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.064208 22561 heartbeater.cc:461] Registering TS with master...
02:12:40.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.065299 22561 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:40.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.068404 22562 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:40.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.068784 22562 heartbeater.cc:461] Registering TS with master...
02:12:40.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.069243 21527 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:40.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.069464 22562 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:40.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.072962 21452 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:40.086 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] handling channelInactive
02:12:40.086 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state READY due to: connection closed
02:12:40.087 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:40.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:40.087463 22571 negotiation.cc:336] Failed RPC negotiation. Trace:
02:12:40.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.034851 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.232.3:58713 (local address 127.20.232.2:46235)
02:12:40.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.038642 (+  3791us) server_negotiation.cc:207] Beginning negotiation
02:12:40.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.038674 (+    32us) server_negotiation.cc:400] Waiting for connection header
02:12:40.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.038744 (+    70us) server_negotiation.cc:408] Connection header received
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.038943 (+   199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.038980 (+    37us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.039356 (+   376us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.049137 (+  9781us) server_negotiation.cc:227] Negotiated authn=SASL
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.050898 (+  1761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.063959 (+ 13061us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:40.085498 (+ 21539us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.232.3:58713: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:12:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3019,"thread_start_us":4566,"threads_started":1}
02:12:40.088 [DEBUG - Time-limited test] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:40.089 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.090 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.090 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.098 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:40.099 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:40.099 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [10ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1231294468(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@6fb7c4e1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21d61530 -> wakeup thread Time-limited test)) after lookup
02:12:40.100 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:40.101 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.101 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.101 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.119 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:40.119 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:40.119 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [10ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [12ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [13ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [13ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [30ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1231294468(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@6fb7c4e1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21d61530 -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057))) after lookup
02:12:40.121 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:40.122 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.122 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.122 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.139 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:40.139 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:40.140 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [10ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [12ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [13ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [13ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [30ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [32ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [34ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [34ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [51ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1231294468(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@6fb7c4e1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21d61530 -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983))) after lookup
02:12:40.141 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:40.142 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.142 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.142 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.259 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:40.259 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:40.259 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=171), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [10ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [12ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [13ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [13ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [30ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [32ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [34ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [34ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [51ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [52ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [54ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [54ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [166ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [170ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1231294468(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@6fb7c4e1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983) -> (continuation of Deferred@1405232434 after retry RPC@868449871), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21d61530 -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983) -> (continuation of Deferred@1405232434 after retry RPC@868449871))) after lookup
02:12:40.260 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:40.262 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.262 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.262 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.280 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:40.280 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:40.280 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=192), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [10ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [12ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [13ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [13ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [30ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [32ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [34ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [34ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [51ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [52ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [54ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [54ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [166ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [170ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [172ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [174ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [174ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [191ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1231294468(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@6fb7c4e1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983) -> (continuation of Deferred@1405232434 after retry RPC@868449871) -> (continuation of Deferred@277174918 after retry RPC@941391839), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21d61530 -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983) -> (continuation of Deferred@1405232434 after retry RPC@868449871) -> (continuation of Deferred@277174918 after retry RPC@941391839))) after lookup
02:12:40.281 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:40.283 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.283 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:40.284 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:41.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.066641 22560 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:41.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.189705 22572 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:12:41.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.190259 22572 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:41.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.192747 22572 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:41.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:41.196780 22448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:41.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:41.201143 22448 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:41.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.203960 22325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "eccfb2228fdd4bc18088d450500f2742" is_pre_election: true
02:12:41.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.205232 22448 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55; no voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742
02:12:41.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.205875 22572 raft_consensus.cc:2749] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
02:12:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.298960 22577 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85d91954824a403e867a4180a90224ea)
02:12:41.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.299443 22577 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:41.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.301995 22577 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:41.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:41.305800 22259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:41.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:41.310170 22259 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:41.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.314312 22514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" is_pre_election: true
02:12:41.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.315018 22514 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 2.
02:12:41.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.316691 22259 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742; no voters: 85d91954824a403e867a4180a90224ea
02:12:41.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.317535 22577 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 3 FOLLOWER]: Leader pre-election won for term 4
02:12:41.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.317883 22577 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85d91954824a403e867a4180a90224ea)
02:12:41.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.318161 22577 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 3 FOLLOWER]: Advancing to term 4
02:12:41.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.323395 22577 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:41.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.325115 22577 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 4 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:41.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.326727 22514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:41.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.327172 22514 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 2 FOLLOWER]: Advancing to term 4
02:12:41.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:41.328680 22259 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:41.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.334906 22514 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 4.
02:12:41.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.336038 22259 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742; no voters: 85d91954824a403e867a4180a90224ea
02:12:41.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.336773 22577 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 4 FOLLOWER]: Leader election won for term 4
02:12:41.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.338366 22577 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 4 LEADER]: Becoming Leader. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Running, Role: LEADER
02:12:41.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.339253 22577 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:41.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.348747 21606 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 reported cstate change: term changed from 3 to 4, leader changed from 85d91954824a403e867a4180a90224ea (127.20.232.3) to eccfb2228fdd4bc18088d450500f2742 (127.20.232.1). New cstate: current_term: 4 leader_uuid: "eccfb2228fdd4bc18088d450500f2742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: UNKNOWN } } }
02:12:41.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.831732 22514 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 4 FOLLOWER]: Refusing update from remote peer eccfb2228fdd4bc18088d450500f2742: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
02:12:41.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.833268 22577 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
02:12:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.846169 22596 mvcc.cc:204] Tried to move back new op lower bound from 7305120800852598784 to 7305120791414775808. Current Snapshot: MvccSnapshot[applied={T|T < 7305120800852598784}]
02:12:41.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:41.881773 22259 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 -> Peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Couldn't send request to peer 85d91954824a403e867a4180a90224ea. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:41.897 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:41.898 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:41.899 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1810), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [10ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [12ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [13ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [13ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [30ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [32ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [34ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [34ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [46ms] refreshing cache from master, [46ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [51ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [52ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [54ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [54ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [166ms] refreshing cache from master, [166ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [170ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [172ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [174ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [174ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [186ms] refreshing cache from master, [186ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [191ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [193ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [196ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [196ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [1805ms] refreshing cache from master, [1806ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [1809ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1231294468(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@6fb7c4e1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983) -> (continuation of Deferred@1405232434 after retry RPC@868449871) -> (continuation of Deferred@277174918 after retry RPC@941391839) -> (continuation of Deferred@743436024 after retry RPC@2004262441), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21d61530 -> wakeup thread Time-limited test -> (continuation of Deferred@371066767 after retry RPC@124846057) -> (continuation of Deferred@1722877575 after retry RPC@1715347983) -> (continuation of Deferred@1405232434 after retry RPC@868449871) -> (continuation of Deferred@277174918 after retry RPC@941391839) -> (continuation of Deferred@743436024 after retry RPC@2004262441))) after lookup
02:12:41.899 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:41.900 [DEBUG - kudu-nio-0] (Connection.java:757) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] Successfully connected to peer
02:12:41.929 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x77fede0b, L:/127.0.0.1:57588 - R:/127.20.232.1:38909] using TOKEN/null
02:12:41.979 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.3:34111
02:12:41.979 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:41.980799 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:34111
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37977
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:41.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:42.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:42.304839 22609 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:42.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:42.305474 22609 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:42.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:42.306443 22609 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:42.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:42.340915 22609 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:42.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:42.341944 22609 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.3
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:42.383764 22609 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:34111
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37977
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:42.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:42.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:42.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:42.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:42.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:42.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:42.385474 22609 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:42.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:42.387787 22609 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:42.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:42.403299 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:42.404253 22617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:43.807694 22615 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 22609
02:12:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:43.860747 22609 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.457s	user 0.499s	sys 0.888s
02:12:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:43.861295 22609 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.458s	user 0.499s	sys 0.889s
02:12:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:43.862586 22619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:43.864500 22618 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1456 milliseconds
02:12:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.864552 22609 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:43.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.865895 22609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:43.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.868105 22609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:43.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.869553 22609 hybrid_clock.cc:648] HybridClock initialized: now 1783476763869498 us; error 58 us; skew 500 ppm
02:12:43.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.870412 22609 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:43.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.877336 22609 webserver.cc:533] Webserver started at http://127.20.232.3:37977/ using document root <none> and password file <none>
02:12:43.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.878381 22609 fs_manager.cc:362] Metadata directory not provided
02:12:43.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.878636 22609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:43.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.887327 22609 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.002s
02:12:43.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.892932 22633 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:43.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.894999 22609 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
02:12:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.895313 22609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85d91954824a403e867a4180a90224ea"
02:12:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:27 on dist-test-slave-h94b"
02:12:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.898144 22609 fs_report.cc:389] FS layout report
02:12:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/data
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:43.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.949536 22609 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:43.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.951288 22609 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:43.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.951694 22609 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:43.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.954339 22609 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:43.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.960245 22640 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:43.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.969408 22609 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:43.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.969663 22609 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
02:12:43.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.969908 22609 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:43.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.975116 22609 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:43.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.975333 22609 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
02:12:43.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:43.976356 22640 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap starting.
02:12:44.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.049556 22640 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Log is configured to *not* fsync() on all Append() calls
02:12:44.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.096356 22640 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.097244 22640 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap complete.
02:12:44.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.098551 22640 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent bootstrapping tablet: real 0.123s	user 0.099s	sys 0.020s
02:12:44.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.119349 22640 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:44.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.121734 22640 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d91954824a403e867a4180a90224ea, State: Initialized, Role: FOLLOWER
02:12:44.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.123073 22640 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:44.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.127049 22640 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent starting tablet: real 0.028s	user 0.018s	sys 0.008s
02:12:44.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.169974 22609 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.3:34111
02:12:44.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.170307 22751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.3:34111 every 8 connection(s)
02:12:44.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.172559 22609 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:44.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.177693 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 22609
02:12:44.179 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:44.203 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:44.210 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.1:38909
02:12:44.211 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:44.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.211719 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 22229
02:12:44.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.226075 22754 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:44.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.226601 22754 heartbeater.cc:461] Registering TS with master...
02:12:44.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.227931 22754 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:44.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.228116 22752 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:44.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.228487 22752 heartbeater.cc:461] Registering TS with master...
02:12:44.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.229401 22752 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:44.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.230850 22753 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:44.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.231191 22753 heartbeater.cc:461] Registering TS with master...
02:12:44.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.232079 22753 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:44.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.232664 21608 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:44.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.233210 21452 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:44.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.235466 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.3:54673
02:12:44.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:44.236270 21527 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:44.244 [DEBUG - kudu-nio-0] (Connection.java:239) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] handling channelInactive
02:12:44.245 [DEBUG - kudu-nio-0] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state READY due to: connection closed
02:12:44.245 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:44.246 [DEBUG - Time-limited test] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.247 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.247 [DEBUG - kudu-nio-1] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.247 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.259 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:44.259 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:44.260 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test)) after lookup
02:12:44.261 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.262 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.262 [DEBUG - kudu-nio-2] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.263 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.278 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:44.279 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:44.279 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [16ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494))) after lookup
02:12:44.280 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.281 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.281 [DEBUG - kudu-nio-3] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.281 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.299 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:44.299 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:44.299 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=54), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [16ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [36ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [53ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800))) after lookup
02:12:44.301 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.302 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.302 [DEBUG - kudu-nio-4] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.302 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.338 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:44.338 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:44.339 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=93), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [16ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [36ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [53ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [55ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [57ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [57ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [93ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125))) after lookup
02:12:44.339 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.340 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.340 [DEBUG - kudu-nio-5] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.341 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.457 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:44.458 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:44.458 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=212), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [16ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [36ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [53ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [55ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [57ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [57ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [93ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [94ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [95ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [96ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [212ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230))) after lookup
02:12:44.459 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.460 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.460 [DEBUG - kudu-nio-6] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.460 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:44.698 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:44.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=453), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [16ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [36ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [53ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [55ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [57ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [57ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [93ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [94ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [95ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [96ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [212ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [214ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [215ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [215ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [449ms] refreshing cache from master, [449ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [453ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230) -> (continuation of Deferred@1415847201 after retry RPC@931494698), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230) -> (continuation of Deferred@1415847201 after retry RPC@931494698))) after lookup
02:12:44.700 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.701 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.701 [DEBUG - kudu-nio-7] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.701 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:44.959 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:44.959 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=713), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [16ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [36ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [53ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [55ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [57ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [57ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [93ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [94ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [95ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [96ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [212ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [214ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [215ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [215ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [449ms] refreshing cache from master, [449ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [453ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [454ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [456ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [456ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [709ms] refreshing cache from master, [709ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [713ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230) -> (continuation of Deferred@1415847201 after retry RPC@931494698) -> (continuation of Deferred@1431210787 after retry RPC@1234101899), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230) -> (continuation of Deferred@1415847201 after retry RPC@931494698) -> (continuation of Deferred@1431210787 after retry RPC@1234101899))) after lookup
02:12:44.960 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:44.961 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.961 [DEBUG - kudu-nio-0] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:44.962 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:45.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.239353 22752 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:45.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.404898 22763 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:12:45.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.405346 22763 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:45.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.407074 22763 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:45.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:45.414414 22448 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:45.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.418502 22705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea" is_pre_election: true
02:12:45.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.419157 22705 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 3.
02:12:45.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.420367 22448 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:12:45.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.420990 22763 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 4 FOLLOWER]: Leader pre-election won for term 5
02:12:45.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.421277 22763 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:12:45.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.421499 22763 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 4 FOLLOWER]: Advancing to term 5
02:12:45.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.426906 22763 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:45.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.428613 22763 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 5 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:45.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.429417 22705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea"
02:12:45.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.429836 22705 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 3 FOLLOWER]: Advancing to term 5
02:12:45.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:45.433358 22448 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:45.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.436827 22705 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 5.
02:12:45.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.437830 22448 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:12:45.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.438596 22763 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 5 FOLLOWER]: Leader election won for term 5
02:12:45.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.439560 22763 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 5 LEADER]: Becoming Leader. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Running, Role: LEADER
02:12:45.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.440407 22763 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:45.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:45.450902 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 reported cstate change: term changed from 4 to 5, leader changed from eccfb2228fdd4bc18088d450500f2742 (127.20.232.1) to 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2). New cstate: current_term: 5 leader_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: HEALTHY } } }
02:12:46.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.048736 22448 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 -> Peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Couldn't send request to peer eccfb2228fdd4bc18088d450500f2742. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:46.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:46.059182 22705 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 5 FOLLOWER]: Refusing update from remote peer 75d6e7b68fdf468cb97ac9fa184bec55: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
02:12:46.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:46.060590 22782 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
02:12:46.359 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:46.359 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:46.359 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=16, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2114), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [16ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [29ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [36ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [36ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [53ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [55ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [57ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [57ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [89ms] refreshing cache from master, [89ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [93ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [94ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [95ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [96ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [209ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [212ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [214ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [215ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [215ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [449ms] refreshing cache from master, [449ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [453ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [454ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [456ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [456ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [709ms] refreshing cache from master, [709ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [713ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [715ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [716ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [717ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2109ms] refreshing cache from master, [2109ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [2113ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1971528587(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@36c76cc4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230) -> (continuation of Deferred@1415847201 after retry RPC@931494698) -> (continuation of Deferred@1431210787 after retry RPC@1234101899) -> (continuation of Deferred@1539548862 after retry RPC@1699195379), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37765cd5 -> wakeup thread Time-limited test -> (continuation of Deferred@1279933866 after retry RPC@1195961494) -> (continuation of Deferred@1960435737 after retry RPC@1962249800) -> (continuation of Deferred@2062919141 after retry RPC@573240125) -> (continuation of Deferred@1660096894 after retry RPC@397780230) -> (continuation of Deferred@1415847201 after retry RPC@931494698) -> (continuation of Deferred@1431210787 after retry RPC@1234101899) -> (continuation of Deferred@1539548862 after retry RPC@1699195379))) after lookup
02:12:46.360 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:46.361 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] Successfully connected to peer
02:12:46.390 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x5e68610f, L:/127.0.0.1:54240 - R:/127.20.232.2:46235] using TOKEN/null
02:12:46.450 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.1:38909
02:12:46.450 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:46.451884 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:46.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:46.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:46.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.623399 22448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111) [suppressed 3 similar messages]
02:12:46.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.768523 22795 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:46.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.769333 22795 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:46.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.769953 22795 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:46.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.805773 22795 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:46.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.806696 22795 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.1
02:12:46.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:46.845600 22795 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:46.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:46.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:46.847092 22795 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:46.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:46.848758 22795 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:46.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.861683 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:46.862356 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:48.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:48.141367 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:48.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:48.142256 22795 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.281s	user 0.451s	sys 0.803s
02:12:48.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:48.142629 22795 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.281s	user 0.451s	sys 0.803s
02:12:48.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:48.142302 22805 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1277 milliseconds
02:12:48.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.142916 22795 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:48.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.144192 22795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:48.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.146411 22795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:48.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.147809 22795 hybrid_clock.cc:648] HybridClock initialized: now 1783476768147759 us; error 57 us; skew 500 ppm
02:12:48.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.148672 22795 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.156643 22795 webserver.cc:533] Webserver started at http://127.20.232.1:33627/ using document root <none> and password file <none>
02:12:48.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.157766 22795 fs_manager.cc:362] Metadata directory not provided
02:12:48.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.158061 22795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:48.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.169658 22795 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.001s	sys 0.005s
02:12:48.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.175900 22816 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:48.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.178663 22795 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.003s	sys 0.003s
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.179502 22795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "eccfb2228fdd4bc18088d450500f2742"
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:22 on dist-test-slave-h94b"
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.181895 22795 fs_report.cc:389] FS layout report
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/data
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:48.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:48.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:48.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:48.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:48.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:48.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:48.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.255703 22795 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:48.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.257289 22795 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:48.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.257794 22795 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:48.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.260481 22795 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:48.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.266290 22823 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:48.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.275033 22795 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:48.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.275295 22795 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
02:12:48.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.275527 22795 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:48.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.282770 22795 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:48.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.283070 22795 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.006s	sys 0.001s
02:12:48.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.283658 22823 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap starting.
02:12:48.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.361270 22823 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Log is configured to *not* fsync() on all Append() calls
02:12:48.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.440876 22823 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:48.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.442102 22823 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap complete.
02:12:48.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.443964 22823 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent bootstrapping tablet: real 0.161s	user 0.131s	sys 0.024s
02:12:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.460582 22795 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.1:38909
02:12:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.460721 22931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.1:38909 every 8 connection(s)
02:12:48.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.465142 22795 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:12:48.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.467593 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 22795
02:12:48.470 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:48.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.471730 22823 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:48.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.475476 22823 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Initialized, Role: FOLLOWER
02:12:48.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.476894 22823 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:48.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.488550 22823 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent starting tablet: real 0.044s	user 0.033s	sys 0.008s
02:12:48.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.521196 22932 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:48.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.521711 22932 heartbeater.cc:461] Registering TS with master...
02:12:48.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.523164 22932 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:48.526 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:48.527 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.2:46235
02:12:48.528 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:48.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.528448 21606 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:48.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.530658 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 22426
02:12:48.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.532748 21606 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.1:37293
02:12:48.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.537487 22932 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:48.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.564443 22938 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:48.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.564957 22938 heartbeater.cc:461] Registering TS with master...
02:12:48.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.565711 22938 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:48.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.568943 22933 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:48.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.569370 22933 heartbeater.cc:461] Registering TS with master...
02:12:48.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.570050 22933 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:48.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.570230 21452 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:48.573513 21527 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:48.584 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] handling channelInactive
02:12:48.584 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:48.585 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state READY due to: connection closed
02:12:48.585 [DEBUG - Time-limited test] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:48.587 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.587 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.587 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.599 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:48.599 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:48.600 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [13ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1453267611(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68051391 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@152b0ad3 -> wakeup thread Time-limited test)) after lookup
02:12:48.601 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:48.602 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.602 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.603 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.618 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:48.618 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:48.618 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [13ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [18ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [18ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [32ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1453267611(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68051391 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@152b0ad3 -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603))) after lookup
02:12:48.619 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:48.620 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.620 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.621 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:48.659 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:48.659 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [13ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [18ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [18ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [32ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [35ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [36ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1453267611(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68051391 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@152b0ad3 -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175))) after lookup
02:12:48.660 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:48.661 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.661 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.662 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.719 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:48.720 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:48.720 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=135), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [13ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [18ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [18ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [32ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [35ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [36ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [74ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [77ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [77ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [129ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [134ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1453267611(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68051391 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175) -> (continuation of Deferred@281474334 after retry RPC@301732575), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@152b0ad3 -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175) -> (continuation of Deferred@281474334 after retry RPC@301732575))) after lookup
02:12:48.721 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:48.722 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.722 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:48.722 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:49.078 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:49.078 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:49.079 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=493), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [13ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [18ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [18ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [32ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [35ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [36ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [74ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [77ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [77ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [129ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [134ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [135ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [137ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [137ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [489ms] refreshing cache from master, [489ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [493ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1453267611(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68051391 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175) -> (continuation of Deferred@281474334 after retry RPC@301732575) -> (continuation of Deferred@799240877 after retry RPC@1589180268), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@152b0ad3 -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175) -> (continuation of Deferred@281474334 after retry RPC@301732575) -> (continuation of Deferred@799240877 after retry RPC@1589180268))) after lookup
02:12:49.080 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:12:49.081 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:49.081 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:49.081 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:12:49.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.681490 22946 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 75d6e7b68fdf468cb97ac9fa184bec55)
02:12:49.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.681893 22946 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:49.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.684792 22946 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:49.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:49.688884 22637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:12:49.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:49.693710 22637 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:12:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.697700 22887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "85d91954824a403e867a4180a90224ea" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "eccfb2228fdd4bc18088d450500f2742" is_pre_election: true
02:12:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.698475 22887 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d91954824a403e867a4180a90224ea in term 4.
02:12:49.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.699739 22637 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742; no voters: 75d6e7b68fdf468cb97ac9fa184bec55
02:12:49.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.700382 22946 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 5 FOLLOWER]: Leader pre-election won for term 6
02:12:49.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.700678 22946 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 5 FOLLOWER]: Starting leader election (detected failure of leader 75d6e7b68fdf468cb97ac9fa184bec55)
02:12:49.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.700979 22946 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 5 FOLLOWER]: Advancing to term 6
02:12:49.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.705672 22946 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.707369 22946 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 6 election: Requested vote from peers eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:49.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.708312 22887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "85d91954824a403e867a4180a90224ea" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "eccfb2228fdd4bc18088d450500f2742"
02:12:49.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.708738 22887 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 4 FOLLOWER]: Advancing to term 6
02:12:49.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:49.712309 22637 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:12:49.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.715673 22887 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d91954824a403e867a4180a90224ea in term 6.
02:12:49.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.716604 22637 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742; no voters: 75d6e7b68fdf468cb97ac9fa184bec55
02:12:49.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.717281 22946 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 6 FOLLOWER]: Leader election won for term 6
02:12:49.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.718989 22946 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 6 LEADER]: Becoming Leader. State: Replica: 85d91954824a403e867a4180a90224ea, State: Running, Role: LEADER
02:12:49.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.719902 22946 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:49.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:49.730365 21606 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea reported cstate change: term changed from 5 to 6, leader changed from 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2) to 85d91954824a403e867a4180a90224ea (127.20.232.3). New cstate: current_term: 6 leader_uuid: "85d91954824a403e867a4180a90224ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: UNKNOWN } } }
02:12:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:50.185904 22887 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 6 FOLLOWER]: Refusing update from remote peer 85d91954824a403e867a4180a90224ea: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
02:12:50.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:50.187469 22946 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
02:12:50.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:50.199209 22637 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea -> Peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Couldn't send request to peer 75d6e7b68fdf468cb97ac9fa184bec55. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:51.038 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:51.038 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:51.039 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2453), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [13ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [18ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [18ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [32ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [34ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [35ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [36ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [69ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [74ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [77ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [77ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [129ms] refreshing cache from master, [130ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [134ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [135ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [137ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [137ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [489ms] refreshing cache from master, [489ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [493ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [494ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [496ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [496ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2449ms] refreshing cache from master, [2449ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [2453ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1453267611(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68051391 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175) -> (continuation of Deferred@281474334 after retry RPC@301732575) -> (continuation of Deferred@799240877 after retry RPC@1589180268) -> (continuation of Deferred@1700421194 after retry RPC@2093008410), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@152b0ad3 -> wakeup thread Time-limited test -> (continuation of Deferred@1610124221 after retry RPC@315129603) -> (continuation of Deferred@191508510 after retry RPC@1519089175) -> (continuation of Deferred@281474334 after retry RPC@301732575) -> (continuation of Deferred@799240877 after retry RPC@1589180268) -> (continuation of Deferred@1700421194 after retry RPC@2093008410))) after lookup
02:12:51.039 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:51.040 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] Successfully connected to peer
02:12:51.061 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x3e5f063e, L:/127.0.0.1:47970 - R:/127.20.232.3:34111] using TOKEN/null
02:12:51.111 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.2:46235
02:12:51.113 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:51.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:51.114502 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:46235
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36149
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:51.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:51.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:51.435949 22980 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:51.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:51.436446 22980 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:51.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:51.436914 22980 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:51.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:51.485724 22980 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:51.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:51.487185 22980 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.2
02:12:51.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:51.538322 22980 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:51.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:51.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:46235
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36149
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:51.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:51.539620 22980 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:51.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:51.541216 22980 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:51.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:51.553460 22986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:52.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:52.546648 22637 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea -> Peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Couldn't send request to peer 75d6e7b68fdf468cb97ac9fa184bec55. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:53.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:52.956837 22985 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 22980
02:12:53.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:53.018565 22985 kernel_stack_watchdog.cc:198] Thread 22980 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
02:12:53.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
02:12:53.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
02:12:53.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
02:12:53.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
02:12:53.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:51.554786 22987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:53.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:53.019821 22980 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.466s	user 0.485s	sys 0.941s
02:12:53.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:53.020406 22980 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.467s	user 0.485s	sys 0.942s
02:12:53.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:53.021461 22989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:53.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:53.026284 22988 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1471 milliseconds
02:12:53.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.026304 22980 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:53.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.027522 22980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:53.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.029699 22980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:53.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.031164 22980 hybrid_clock.cc:648] HybridClock initialized: now 1783476773031134 us; error 35 us; skew 500 ppm
02:12:53.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.032328 22980 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.040438 22980 webserver.cc:533] Webserver started at http://127.20.232.2:36149/ using document root <none> and password file <none>
02:12:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.041651 22980 fs_manager.cc:362] Metadata directory not provided
02:12:53.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.041908 22980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:53.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.051359 22980 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.002s	sys 0.002s
02:12:53.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.056214 22999 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:53.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.057323 22980 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.003s
02:12:53.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.057687 22980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:53.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:25 on dist-test-slave-h94b"
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.059813 22980 fs_report.cc:389] FS layout report
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/data
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:53.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:53.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.113605 22980 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:53.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.115341 22980 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:53.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.115877 22980 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:53.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.118637 22980 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:53.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.124951 23006 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:53.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.135448 22980 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:53.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.135715 22980 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.001s
02:12:53.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.136045 22980 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.141105 22980 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.141340 22980 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.000s	sys 0.003s
02:12:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.142218 23006 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap starting.
02:12:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.210673 23006 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Log is configured to *not* fsync() on all Append() calls
02:12:53.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.297258 23006 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.298839 23006 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap complete.
02:12:53.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.301043 23006 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent bootstrapping tablet: real 0.159s	user 0.121s	sys 0.031s
02:12:53.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.327607 23006 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:53.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.330950 23006 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Initialized, Role: FOLLOWER
02:12:53.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.331694 22980 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.2:46235
02:12:53.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.332202 23115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.2:46235 every 8 connection(s)
02:12:53.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.332163 23006 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:53.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.335316 22980 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:12:53.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.335856 23006 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent starting tablet: real 0.034s	user 0.029s	sys 0.003s
02:12:53.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.336144 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 22980
02:12:53.337 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:53.363 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:53.364 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.3:34111
02:12:53.365 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:53.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.365422 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 22609
02:12:53.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.371701 23117 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:53.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.372189 23117 heartbeater.cc:461] Registering TS with master...
02:12:53.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.373317 23117 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:53.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.375854 23118 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:53.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.376210 23118 heartbeater.cc:461] Registering TS with master...
02:12:53.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.377089 23118 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:53.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.377714 21608 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:53.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.381240 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.2:35215
02:12:53.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.381187 21526 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:53.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.386140 23119 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:53.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.386518 23119 heartbeater.cc:461] Registering TS with master...
02:12:53.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.387121 23119 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:53.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:53.390623 21452 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:53.405 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] handling channelInactive
02:12:53.405 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state READY due to: connection closed
02:12:53.406 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:53.407 [DEBUG - Time-limited test] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:53.409 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.409 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.409 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.420 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:53.420 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:53.421 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [3ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@742258375(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5637154c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2fc673a2 -> wakeup thread Time-limited test)) after lookup
02:12:53.422 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:53.423 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.423 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.423 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.440 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:53.441 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:53.441 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [3ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [17ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [17ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@742258375(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5637154c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2fc673a2 -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729))) after lookup
02:12:53.442 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:53.443 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.443 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.443 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.479 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:53.480 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:53.480 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [3ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [17ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [17ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [35ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [37ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [37ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@742258375(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5637154c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2fc673a2 -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392))) after lookup
02:12:53.480 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:53.482 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.482 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.482 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.558 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:53.559 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:53.559 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=152), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [3ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [17ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [17ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [35ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [37ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [37ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [74ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [76ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [76ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [148ms] refreshing cache from master, [149ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [152ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@742258375(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5637154c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392) -> (continuation of Deferred@2116005529 after retry RPC@866279165), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2fc673a2 -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392) -> (continuation of Deferred@2116005529 after retry RPC@866279165))) after lookup
02:12:53.560 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:53.561 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.561 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.561 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:53.958 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:12:53.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=552), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [3ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [17ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [17ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [35ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [37ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [37ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [74ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [76ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [76ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [148ms] refreshing cache from master, [149ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [152ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [153ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [155ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [155ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [548ms] refreshing cache from master, [548ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [552ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@742258375(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5637154c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392) -> (continuation of Deferred@2116005529 after retry RPC@866279165) -> (continuation of Deferred@379748523 after retry RPC@936796468), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2fc673a2 -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392) -> (continuation of Deferred@2116005529 after retry RPC@866279165) -> (continuation of Deferred@379748523 after retry RPC@936796468))) after lookup
02:12:53.959 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:12:53.960 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.960 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:53.960 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:12:54.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.386279 23117 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:54.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.546461 23131 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85d91954824a403e867a4180a90224ea)
02:12:54.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.546970 23131 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:54.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.549291 23131 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:54.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:54.553419 22820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:54.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:54.556774 22820 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:54.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.562176 23071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" is_pre_election: true
02:12:54.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.563014 23071 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 5.
02:12:54.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.564468 22820 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742; no voters: 85d91954824a403e867a4180a90224ea
02:12:54.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.565337 23131 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 6 FOLLOWER]: Leader pre-election won for term 7
02:12:54.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.565759 23131 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 85d91954824a403e867a4180a90224ea)
02:12:54.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.566170 23131 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 6 FOLLOWER]: Advancing to term 7
02:12:54.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.571019 23131 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:54.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.572590 23131 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 7 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:12:54.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.574704 23071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:12:54.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.575268 23071 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 5 FOLLOWER]: Advancing to term 7
02:12:54.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:54.576300 22820 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:12:54.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.581923 23071 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 7.
02:12:54.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.582975 22820 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742; no voters: 85d91954824a403e867a4180a90224ea
02:12:54.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.583593 23131 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 7 FOLLOWER]: Leader election won for term 7
02:12:54.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.585403 23131 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 7 LEADER]: Becoming Leader. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Running, Role: LEADER
02:12:54.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.586752 23131 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:54.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.595146 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 reported cstate change: term changed from 6 to 7, leader changed from 85d91954824a403e867a4180a90224ea (127.20.232.3) to eccfb2228fdd4bc18088d450500f2742 (127.20.232.1). New cstate: current_term: 7 leader_uuid: "eccfb2228fdd4bc18088d450500f2742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: UNKNOWN } } }
02:12:54.878 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:54.878 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:54.879 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1472), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [3ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [8ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [14ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [15ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [17ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [17ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [33ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [35ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [37ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [37ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [73ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [74ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [76ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [76ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [148ms] refreshing cache from master, [149ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [152ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [153ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [155ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [155ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [548ms] refreshing cache from master, [548ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [552ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [553ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [554ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [555ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [1468ms] refreshing cache from master, [1468ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [1472ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@742258375(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@5637154c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392) -> (continuation of Deferred@2116005529 after retry RPC@866279165) -> (continuation of Deferred@379748523 after retry RPC@936796468) -> (continuation of Deferred@971271867 after retry RPC@1035954034), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2fc673a2 -> wakeup thread Time-limited test -> (continuation of Deferred@550361616 after retry RPC@382309729) -> (continuation of Deferred@2100670450 after retry RPC@663572392) -> (continuation of Deferred@2116005529 after retry RPC@866279165) -> (continuation of Deferred@379748523 after retry RPC@936796468) -> (continuation of Deferred@971271867 after retry RPC@1035954034))) after lookup
02:12:54.880 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:54.881 [DEBUG - kudu-nio-7] (Connection.java:757) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] Successfully connected to peer
02:12:54.910 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x5777ebc6, L:/127.0.0.1:55992 - R:/127.20.232.1:38909] using TOKEN/null
02:12:54.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.919245 23071 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 7 FOLLOWER]: Refusing update from remote peer eccfb2228fdd4bc18088d450500f2742: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 26. (index mismatch)
02:12:54.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.921249 23131 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
02:12:54.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:54.921835 22820 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 -> Peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Couldn't send request to peer 85d91954824a403e867a4180a90224ea. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:54.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.936053 23152 mvcc.cc:204] Tried to move back new op lower bound from 7305120854285103104 to 7305120848780451840. Current Snapshot: MvccSnapshot[applied={T|T < 7305120854285103104}]
02:12:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.944424 23150 mvcc.cc:204] Tried to move back new op lower bound from 7305120870054834176 to 7305120868714377216. Current Snapshot: MvccSnapshot[applied={T|T < 7305120870054834176}]
02:12:54.984 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.3:34111
02:12:54.984 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:54.986835 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:34111
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:12:54.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:54.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37977
02:12:54.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:12:54.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:54.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:54.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:54.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:55.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:55.297463 23158 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:55.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:55.298040 23158 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:55.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:55.298656 23158 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:55.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:55.332757 23158 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:55.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:55.333874 23158 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.3
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:55.372718 23158 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:34111
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37977
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:12:55.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:55.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:55.374442 23158 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:55.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:55.376247 23158 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:12:55.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:55.389406 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:56.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:55.390568 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:56.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:56.602414 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:56.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:56.604676 23166 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1210 milliseconds
02:12:56.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.604823 23158 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:56.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.605965 23158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:56.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.608227 23158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:56.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.609609 23158 hybrid_clock.cc:648] HybridClock initialized: now 1783476776609567 us; error 58 us; skew 500 ppm
02:12:56.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.610508 23158 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:56.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.616834 23158 webserver.cc:533] Webserver started at http://127.20.232.3:37977/ using document root <none> and password file <none>
02:12:56.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.617811 23158 fs_manager.cc:362] Metadata directory not provided
02:12:56.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.618077 23158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:56.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.626325 23158 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.001s
02:12:56.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.631182 23177 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.632411 23158 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
02:12:56.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.632864 23158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:56.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85d91954824a403e867a4180a90224ea"
02:12:56.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:27 on dist-test-slave-h94b"
02:12:56.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.634953 23158 fs_report.cc:389] FS layout report
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/data
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:56.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:56.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.691646 23158 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:56.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.693413 23158 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:56.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.693866 23158 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:56.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.696569 23158 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:56.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.703094 23184 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:56.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.712304 23158 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:56.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.712601 23158 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
02:12:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.712917 23158 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:56.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.719647 23158 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:56.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.719866 23158 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.006s	sys 0.000s
02:12:56.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.720417 23184 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap starting.
02:12:56.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.785439 23184 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Log is configured to *not* fsync() on all Append() calls
02:12:56.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.866160 23184 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:56.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.867049 23184 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap complete.
02:12:56.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.868446 23184 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent bootstrapping tablet: real 0.149s	user 0.106s	sys 0.038s
02:12:56.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.886415 23184 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:56.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.888614 23184 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d91954824a403e867a4180a90224ea, State: Initialized, Role: FOLLOWER
02:12:56.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.889624 23184 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.894557 23184 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent starting tablet: real 0.026s	user 0.027s	sys 0.001s
02:12:56.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.905498 23158 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.3:34111
02:12:56.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.905746 23293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.3:34111 every 8 connection(s)
02:12:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.908058 23158 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:12:56.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.917972 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 23158
02:12:56.919 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:56.955 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:56.956 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.1:38909
02:12:56.956 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:56.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.957011 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 22795
02:12:56.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.968014 23294 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:12:56.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.968530 23294 heartbeater.cc:461] Registering TS with master...
02:12:56.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.969849 23294 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:12:56.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.972515 23296 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:12:56.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.972878 23296 heartbeater.cc:461] Registering TS with master...
02:12:56.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.973815 23296 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:12:56.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.977689 21452 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:56.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.979132 21608 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:56.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.979692 23295 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:12:56.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.980046 23295 heartbeater.cc:461] Registering TS with master...
02:12:56.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.980875 23295 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:12:56.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.981756 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.3:44997
02:12:56.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:56.984473 21527 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:12:56.991 [DEBUG - kudu-nio-7] (Connection.java:239) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] handling channelInactive
02:12:56.992 [DEBUG - kudu-nio-7] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state READY due to: connection closed
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:56.993032 23306 negotiation.cc:336] Failed RPC negotiation. Trace:
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.956049 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.232.1:43917 (local address 127.20.232.3:34111)
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.960584 (+  4535us) server_negotiation.cc:207] Beginning negotiation
02:12:56.993 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.960614 (+    30us) server_negotiation.cc:400] Waiting for connection header
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.960664 (+    50us) server_negotiation.cc:408] Connection header received
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.960840 (+   176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.960870 (+    30us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.961233 (+   363us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.961470 (+   237us) server_negotiation.cc:227] Negotiated authn=SASL
02:12:56.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:56.991289 (+ 29819us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.232.1:43917: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:12:56.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3870,"thread_start_us":3484,"threads_started":1}
02:12:56.994 [DEBUG - Time-limited test] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:56.995 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:56.995 [DEBUG - kudu-nio-0] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:56.996 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:57.019 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:57.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [24ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1086973649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68b8d4d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4262152b -> wakeup thread Time-limited test)) after lookup
02:12:57.020 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:57.021 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.021 [DEBUG - kudu-nio-1] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.022 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.038 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:57.038 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:57.038 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [24ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [28ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [28ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [44ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1086973649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68b8d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4262152b -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728))) after lookup
02:12:57.039 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:57.040 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.040 [DEBUG - kudu-nio-2] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.040 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.058 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:57.058 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:57.058 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=65), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [24ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [28ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [28ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [44ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [45ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [46ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [46ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [63ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1086973649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68b8d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4262152b -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956))) after lookup
02:12:57.059 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:57.060 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.060 [DEBUG - kudu-nio-3] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.061 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.139 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:57.140 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:57.140 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=146), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [24ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [28ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [28ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [44ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [45ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [46ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [46ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [63ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [65ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [67ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [67ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [145ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1086973649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68b8d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4262152b -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997))) after lookup
02:12:57.141 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:57.142 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.143 [DEBUG - kudu-nio-4] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.143 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.378 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:57.378 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:57.378 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=384), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [24ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [28ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [28ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [44ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [45ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [46ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [46ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [63ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [65ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [67ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [67ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [145ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [146ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [149ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [149ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [380ms] refreshing cache from master, [380ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [383ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1086973649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68b8d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997) -> (continuation of Deferred@1467074250 after retry RPC@59874106), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4262152b -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997) -> (continuation of Deferred@1467074250 after retry RPC@59874106))) after lookup
02:12:57.380 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:57.381 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.381 [DEBUG - kudu-nio-5] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.381 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.418 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:12:57.418 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:12:57.418 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=424), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [24ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [28ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [28ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [44ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [45ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [46ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [46ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [63ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [65ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [67ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [67ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [145ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [146ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [149ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [149ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [380ms] refreshing cache from master, [380ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [383ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [385ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [387ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [387ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [419ms] refreshing cache from master, [420ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [423ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1086973649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68b8d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997) -> (continuation of Deferred@1467074250 after retry RPC@59874106) -> (continuation of Deferred@1153150464 after retry RPC@1274852838), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4262152b -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997) -> (continuation of Deferred@1467074250 after retry RPC@59874106) -> (continuation of Deferred@1153150464 after retry RPC@1274852838))) after lookup
02:12:57.419 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:12:57.420 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.420 [DEBUG - kudu-nio-6] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.420 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:12:57.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:57.985584 23294 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:12:58.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.238438 23308 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:12:58.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.238842 23308 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:58.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.241075 23308 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:58.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:58.245302 23003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:58.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:58.249715 23003 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:58.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.254195 23248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea" is_pre_election: true
02:12:58.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.254900 23248 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 6.
02:12:58.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.256140 23003 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:12:58.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.256830 23308 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 7 FOLLOWER]: Leader pre-election won for term 8
02:12:58.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.257129 23308 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:12:58.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.257454 23308 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 7 FOLLOWER]: Advancing to term 8
02:12:58.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.262099 23308 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:58.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.263746 23308 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 8 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:12:58.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.264751 23248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea"
02:12:58.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.265234 23248 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 6 FOLLOWER]: Advancing to term 8
02:12:58.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:58.268121 23003 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:12:58.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.271847 23248 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 8.
02:12:58.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.272922 23003 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:12:58.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.273582 23308 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 8 FOLLOWER]: Leader election won for term 8
02:12:58.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.275192 23308 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 8 LEADER]: Becoming Leader. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Running, Role: LEADER
02:12:58.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.276100 23308 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:12:58.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.285372 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 reported cstate change: term changed from 7 to 8, leader changed from eccfb2228fdd4bc18088d450500f2742 (127.20.232.1) to 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2). New cstate: current_term: 8 leader_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: HEALTHY } } }
02:12:58.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.705175 23248 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 8 FOLLOWER]: Refusing update from remote peer 75d6e7b68fdf468cb97ac9fa184bec55: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
02:12:58.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:58.706815 23308 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
02:12:58.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:58.740264 23003 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 -> Peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Couldn't send request to peer eccfb2228fdd4bc18088d450500f2742. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:13:01.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.361014 23003 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 -> Peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Couldn't send request to peer eccfb2228fdd4bc18088d450500f2742. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:13:01.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:01.498 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:01.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4505), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [2ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [24ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [25ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [28ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [28ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [44ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [45ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [46ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [46ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [63ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [65ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [67ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [67ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [140ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [145ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [146ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [149ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [149ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [380ms] refreshing cache from master, [380ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [383ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [385ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [387ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [387ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [419ms] refreshing cache from master, [420ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [423ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [425ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [426ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [426ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [4500ms] refreshing cache from master, [4500ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [4504ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@1086973649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@68b8d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997) -> (continuation of Deferred@1467074250 after retry RPC@59874106) -> (continuation of Deferred@1153150464 after retry RPC@1274852838) -> (continuation of Deferred@800170781 after retry RPC@30171272), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4262152b -> wakeup thread Time-limited test -> (continuation of Deferred@1534326256 after retry RPC@157972728) -> (continuation of Deferred@1662135707 after retry RPC@1996688956) -> (continuation of Deferred@1744182209 after retry RPC@141269997) -> (continuation of Deferred@1467074250 after retry RPC@59874106) -> (continuation of Deferred@1153150464 after retry RPC@1274852838) -> (continuation of Deferred@800170781 after retry RPC@30171272))) after lookup
02:13:01.500 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:01.501 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] Successfully connected to peer
02:13:01.525 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x3c3a7dfd, L:/127.0.0.1:45972 - R:/127.20.232.2:46235] using TOKEN/null
02:13:01.586 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.1:38909
02:13:01.587 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:13:01.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:01.588456 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:01.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:01.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:01.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:13:01.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.897049 23352 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:13:01.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.897543 23352 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:13:01.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.898080 23352 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:13:01.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.931342 23352 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:13:01.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.932340 23352 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.1
02:13:01.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:01.969256 23352 tablet_server_runner.cc:78] Tablet server non-default flags:
02:13:01.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:13:01.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:13:01.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:13:01.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:01.971029 23352 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:13:01.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:01.972728 23352 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:13:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.985409 23358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:03.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.986788 23359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:03.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:01.989777 23361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:03.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:03.117762 23360 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
02:13:03.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.117825 23352 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:13:03.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.121956 23352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:13:03.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.124516 23352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:13:03.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.126098 23352 hybrid_clock.cc:648] HybridClock initialized: now 1783476783125939 us; error 54 us; skew 500 ppm
02:13:03.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.127331 23352 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:13:03.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.134727 23352 webserver.cc:533] Webserver started at http://127.20.232.1:33627/ using document root <none> and password file <none>
02:13:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.135824 23352 fs_manager.cc:362] Metadata directory not provided
02:13:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.136082 23352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:13:03.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.144105 23352 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
02:13:03.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.149736 23372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:13:03.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.150987 23352 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
02:13:03.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.151347 23352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:03.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "eccfb2228fdd4bc18088d450500f2742"
02:13:03.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:22 on dist-test-slave-h94b"
02:13:03.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.153366 23352 fs_report.cc:389] FS layout report
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/data
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:13:03.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:13:03.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.201326 23352 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:13:03.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.203141 23352 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:13:03.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.203666 23352 kserver.cc:163] Server-wide thread pool size limit: 3276
02:13:03.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.206593 23352 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:13:03.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.213207 23379 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:13:03.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.225706 23352 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:13:03.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.226161 23352 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.015s	user 0.002s	sys 0.001s
02:13:03.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.226441 23352 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:13:03.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.232589 23352 ts_tablet_manager.cc:616] Registered 1 tablets
02:13:03.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.232803 23352 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.000s	sys 0.004s
02:13:03.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.233687 23379 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap starting.
02:13:03.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.316103 23379 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Log is configured to *not* fsync() on all Append() calls
02:13:03.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.435114 23379 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:13:03.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.436136 23379 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap complete.
02:13:03.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.437879 23379 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent bootstrapping tablet: real 0.205s	user 0.155s	sys 0.035s
02:13:03.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.461107 23379 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:03.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.464591 23379 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Initialized, Role: FOLLOWER
02:13:03.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.466001 23379 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:03.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.484819 23379 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent starting tablet: real 0.047s	user 0.041s	sys 0.003s
02:13:03.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:03.529150 23003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111) [suppressed 10 similar messages]
02:13:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.553784 23352 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.1:38909
02:13:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.553962 23491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.1:38909 every 8 connection(s)
02:13:03.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.557088 23352 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:13:03.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.565282 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 23352
02:13:03.566 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:13:03.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.607908 23494 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:13:03.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.608430 23494 heartbeater.cc:461] Registering TS with master...
02:13:03.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.609764 23494 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:13:03.610 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:13:03.614 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.2:46235
02:13:03.614 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:13:03.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.611253 23492 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:13:03.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.611588 23492 heartbeater.cc:461] Registering TS with master...
02:13:03.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.612506 23492 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:13:03.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.614512 21452 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:13:03.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.616222 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 22980
02:13:03.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.620285 21608 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:13:03.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.623559 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.1:53443
02:13:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.636162 23493 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:13:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.636561 23493 heartbeater.cc:461] Registering TS with master...
02:13:03.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.637207 23493 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:13:03.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:03.640926 21526 ts_manager.cc:194] Re-registered known tserver with Master: eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:13:03.652 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] handling channelInactive
02:13:03.652 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state READY due to: connection closed
02:13:03.653 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:13:03.654 [DEBUG - Time-limited test] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:03.655 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.655 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.655 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.678 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:03.678 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:03.678 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [25ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@844389366(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@507ea514 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@534d8b71 -> wakeup thread Time-limited test)) after lookup
02:13:03.679 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:03.680 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.680 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.681 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:03.699 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:03.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [25ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [26ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [27ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [45ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@844389366(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@507ea514 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@534d8b71 -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156))) after lookup
02:13:03.699 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:03.700 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.700 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.700 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.739 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:03.739 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:03.739 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=86), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [25ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [26ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [27ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [45ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [46ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [47ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [48ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [85ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@844389366(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@507ea514 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@534d8b71 -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090))) after lookup
02:13:03.740 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:03.741 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.741 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.741 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.859 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:03.859 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:03.859 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=206), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [25ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [26ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [27ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [45ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [46ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [47ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [48ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [85ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [87ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [88ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [88ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [201ms] refreshing cache from master, [202ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [205ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@844389366(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@507ea514 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090) -> (continuation of Deferred@2063138452 after retry RPC@372414385), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@534d8b71 -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090) -> (continuation of Deferred@2063138452 after retry RPC@372414385))) after lookup
02:13:03.860 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:03.861 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.861 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:03.861 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:04.078 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:04.078 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:04.079 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=425), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [25ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [26ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [27ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [45ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [46ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [47ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [48ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [85ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [87ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [88ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [88ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [201ms] refreshing cache from master, [202ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [205ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [207ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [208ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [208ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [421ms] refreshing cache from master, [421ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [424ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@844389366(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@507ea514 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090) -> (continuation of Deferred@2063138452 after retry RPC@372414385) -> (continuation of Deferred@474038316 after retry RPC@508351242), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@534d8b71 -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090) -> (continuation of Deferred@2063138452 after retry RPC@372414385) -> (continuation of Deferred@474038316 after retry RPC@508351242))) after lookup
02:13:04.079 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:04.081 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:04.081 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state CONNECTING due to: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:04.082 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235
02:13:04.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.629791 23492 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:13:04.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.766764 23505 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 75d6e7b68fdf468cb97ac9fa184bec55)
02:13:04.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.767194 23505 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:04.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.769469 23505 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:13:04.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:04.775841 23181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:13:04.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:04.781450 23181 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:13:04.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.783665 23444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "85d91954824a403e867a4180a90224ea" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "eccfb2228fdd4bc18088d450500f2742" is_pre_election: true
02:13:04.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.784444 23444 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d91954824a403e867a4180a90224ea in term 7.
02:13:04.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.785725 23181 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742; no voters: 75d6e7b68fdf468cb97ac9fa184bec55
02:13:04.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.786867 23505 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 8 FOLLOWER]: Leader pre-election won for term 9
02:13:04.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.787163 23505 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 8 FOLLOWER]: Starting leader election (detected failure of leader 75d6e7b68fdf468cb97ac9fa184bec55)
02:13:04.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.787510 23505 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 8 FOLLOWER]: Advancing to term 9
02:13:04.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.793099 23505 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:04.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.794648 23505 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 9 election: Requested vote from peers eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:13:04.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.795914 23444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "85d91954824a403e867a4180a90224ea" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "eccfb2228fdd4bc18088d450500f2742"
02:13:04.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.796536 23444 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 7 FOLLOWER]: Advancing to term 9
02:13:04.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:04.798892 23181 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111)
02:13:04.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.804623 23444 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d91954824a403e867a4180a90224ea in term 9.
02:13:04.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.805612 23181 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85d91954824a403e867a4180a90224ea, eccfb2228fdd4bc18088d450500f2742; no voters: 75d6e7b68fdf468cb97ac9fa184bec55
02:13:04.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.806245 23505 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 9 FOLLOWER]: Leader election won for term 9
02:13:04.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.807291 23505 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 9 LEADER]: Becoming Leader. State: Replica: 85d91954824a403e867a4180a90224ea, State: Running, Role: LEADER
02:13:04.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.808099 23505 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:04.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:04.817022 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea reported cstate change: term changed from 8 to 9, leader changed from 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2) to 85d91954824a403e867a4180a90224ea (127.20.232.3). New cstate: current_term: 9 leader_uuid: "85d91954824a403e867a4180a90224ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: UNKNOWN } } }
02:13:05.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:05.192631 23181 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea -> Peer 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235): Couldn't send request to peer 75d6e7b68fdf468cb97ac9fa184bec55. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.2:46235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:13:05.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:05.323575 23444 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 9 FOLLOWER]: Refusing update from remote peer 85d91954824a403e867a4180a90224ea: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
02:13:05.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:05.325313 23505 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.001s
02:13:05.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:05.699 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:05.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2046), Traces: [0ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [2ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [25ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [26ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [27ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [28ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [45ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [46ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [47ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [48ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [85ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [87ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [88ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [88ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [201ms] refreshing cache from master, [202ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [205ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [207ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [208ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [208ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [421ms] refreshing cache from master, [421ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [424ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [426ms] sending RPC to server 75d6e7b68fdf468cb97ac9fa184bec55, [428ms] received response from server 75d6e7b68fdf468cb97ac9fa184bec55: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [429ms] delaying RPC due to: Network error: Failed to connect to peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235): Connection refused: /127.20.232.2:46235, [2041ms] refreshing cache from master, [2041ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [2045ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@844389366(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@507ea514 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090) -> (continuation of Deferred@2063138452 after retry RPC@372414385) -> (continuation of Deferred@474038316 after retry RPC@508351242) -> (continuation of Deferred@2065280192 after retry RPC@53863466), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@534d8b71 -> wakeup thread Time-limited test -> (continuation of Deferred@197808591 after retry RPC@878921156) -> (continuation of Deferred@1105924950 after retry RPC@1411842090) -> (continuation of Deferred@2063138452 after retry RPC@372414385) -> (continuation of Deferred@474038316 after retry RPC@508351242) -> (continuation of Deferred@2065280192 after retry RPC@53863466))) after lookup
02:13:05.700 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:13:05.701 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] Successfully connected to peer
02:13:05.733 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0xb87fbb41, L:/127.0.0.1:45266 - R:/127.20.232.3:34111] using TOKEN/null
02:13:05.784 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.2:46235
02:13:05.784 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:05.785952 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:05.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:46235
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36149
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:05.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:13:06.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.107806 23536 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:13:06.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.108291 23536 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:13:06.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.108745 23536 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:13:06.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.144331 23536 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:13:06.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.145303 23536 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.2
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:06.185736 23536 tablet_server_runner.cc:78] Tablet server non-default flags:
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.2:46235
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.2
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36149
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.2
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/logs
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:13:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:13:06.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:13:06.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:13:06.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:13:06.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:13:06.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:06.187438 23536 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:13:06.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:06.189723 23536 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:13:06.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.203449 23543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:07.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.204545 23544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:07.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:06.207428 23546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:07.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:07.317925 23545 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
02:13:07.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.318112 23536 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:13:07.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.321887 23536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:13:07.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.324285 23536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:13:07.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.325676 23536 hybrid_clock.cc:648] HybridClock initialized: now 1783476787325638 us; error 50 us; skew 500 ppm
02:13:07.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.326546 23536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:13:07.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.333258 23536 webserver.cc:533] Webserver started at http://127.20.232.2:36149/ using document root <none> and password file <none>
02:13:07.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.334281 23536 fs_manager.cc:362] Metadata directory not provided
02:13:07.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.334491 23536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:13:07.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.342937 23536 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
02:13:07.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.348863 23555 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:13:07.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.350696 23536 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
02:13:07.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.351161 23536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:13:07.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:13:07.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:25 on dist-test-slave-h94b"
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.353470 23536 fs_report.cc:389] FS layout report
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/wal
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/data
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:13:07.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:13:07.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:13:07.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:13:07.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.412817 23536 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:13:07.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.414371 23536 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:13:07.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.414803 23536 kserver.cc:163] Server-wide thread pool size limit: 3276
02:13:07.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.417546 23536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:13:07.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.423558 23562 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:13:07.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.432557 23536 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:13:07.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.432813 23536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.000s
02:13:07.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.433041 23536 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:13:07.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.438535 23536 ts_tablet_manager.cc:616] Registered 1 tablets
02:13:07.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.438764 23536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.003s	sys 0.003s
02:13:07.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.439631 23562 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap starting.
02:13:07.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.522305 23562 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Log is configured to *not* fsync() on all Append() calls
02:13:07.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.632685 23562 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:13:07.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.633500 23562 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Bootstrap complete.
02:13:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.634789 23562 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent bootstrapping tablet: real 0.196s	user 0.155s	sys 0.035s
02:13:07.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.640525 23536 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.2:46235
02:13:07.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.640681 23669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.2:46235 every 8 connection(s)
02:13:07.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.643422 23536 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-1/data/info.pb
02:13:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.652609 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 23536
02:13:07.653 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:13:07.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.653759 23562 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:07.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.656996 23562 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Initialized, Role: FOLLOWER
02:13:07.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.657976 23562 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:07.695 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:13:07.696 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.3:34111
02:13:07.696 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:13:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.696827 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 23158
02:13:07.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.711870 23562 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55: Time spent starting tablet: real 0.077s	user 0.044s	sys 0.022s
02:13:07.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.714114 23670 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:13:07.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.714612 23670 heartbeater.cc:461] Registering TS with master...
02:13:07.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.715938 23670 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:13:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.718497 23672 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:13:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.718961 23672 heartbeater.cc:461] Registering TS with master...
02:13:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.720255 23672 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:13:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.721289 21608 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:13:07.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.724263 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.2:60091
02:13:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.727022 21452 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:13:07.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.728189 23670 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:13:07.735 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] handling channelInactive
02:13:07.735 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state READY due to: connection closed
02:13:07.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.737721 23671 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:13:07.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.738132 23671 heartbeater.cc:461] Registering TS with master...
02:13:07.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.738814 23671 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:07.741480 23679 negotiation.cc:336] Failed RPC negotiation. Trace:
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.663506 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.232.3:53219 (local address 127.20.232.2:46235)
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.674675 (+ 11169us) server_negotiation.cc:207] Beginning negotiation
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.674718 (+    43us) server_negotiation.cc:400] Waiting for connection header
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.674813 (+    95us) server_negotiation.cc:408] Connection header received
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.675221 (+   408us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.675259 (+    38us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.675768 (+   509us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.685640 (+  9872us) server_negotiation.cc:227] Negotiated authn=SASL
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.686289 (+   649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.688691 (+  2402us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.701682 (+ 12991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.703049 (+  1367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
02:13:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.706351 (+  3302us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.707976 (+  1625us) server_negotiation.cc:882] Received SASL_INITIATE request from client
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.708020 (+    44us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.713757 (+  5737us) server_negotiation.cc:923] calling sasl_server_start()
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.713878 (+   121us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.714000 (+   122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.714207 (+   207us) server_negotiation.cc:1036] Waiting for connection context
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:07.734255 (+ 20048us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.232.3:53219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10393,"thread_start_us":10006,"threads_started":1}
02:13:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:07.742527 21527 ts_manager.cc:194] Re-registered known tserver with Master: 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:13:07.750 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:13:07.751 [DEBUG - Time-limited test] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:13:07.752 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.752 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.752 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.758 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:07.758 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:07.759 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=8), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [7ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@144111092(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@2e6a55b1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@70a20d27 -> wakeup thread Time-limited test)) after lookup
02:13:07.759 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:13:07.760 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.760 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.760 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.778 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:07.778 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:07.779 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [7ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [9ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [10ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [10ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@144111092(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@2e6a55b1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@70a20d27 -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128))) after lookup
02:13:07.779 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:13:07.780 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.780 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.780 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.798 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:07.798 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:07.799 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [7ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [9ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [10ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [10ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [29ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [30ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [31ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@144111092(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@2e6a55b1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@70a20d27 -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421))) after lookup
02:13:07.799 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:13:07.800 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.800 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.801 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.858 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:07.858 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:07.858 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=108), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [7ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [9ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [10ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [10ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [29ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [30ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [31ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [49ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [51ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [51ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [108ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@144111092(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@2e6a55b1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421) -> (continuation of Deferred@1995917516 after retry RPC@1447266670), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@70a20d27 -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421) -> (continuation of Deferred@1995917516 after retry RPC@1447266670))) after lookup
02:13:07.859 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:13:07.861 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.861 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:07.861 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:08.280 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:08.280 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111)[L],eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:08.280 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=530), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [7ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [9ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [10ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [10ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [29ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [30ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [31ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [49ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [51ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [51ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [108ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [109ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [111ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [111ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [524ms] refreshing cache from master, [524ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [529ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@144111092(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@2e6a55b1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421) -> (continuation of Deferred@1995917516 after retry RPC@1447266670) -> (continuation of Deferred@488569335 after retry RPC@1474038723), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@70a20d27 -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421) -> (continuation of Deferred@1995917516 after retry RPC@1447266670) -> (continuation of Deferred@488569335 after retry RPC@1474038723))) after lookup
02:13:08.281 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] connecting to peer
02:13:08.282 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:08.282 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:08.283 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 85d91954824a403e867a4180a90224ea(127.20.232.3:34111) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111
02:13:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.112818 23684 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 85d91954824a403e867a4180a90224ea)
02:13:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.113225 23684 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:09.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.115554 23684 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:13:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.119467 23376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:13:09.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.123239 23376 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:13:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.128036 23625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" is_pre_election: true
02:13:09.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.128851 23625 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 8.
02:13:09.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.130393 23376 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742; no voters: 85d91954824a403e867a4180a90224ea
02:13:09.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.131263 23684 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 9 FOLLOWER]: Leader pre-election won for term 10
02:13:09.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.131556 23684 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 85d91954824a403e867a4180a90224ea)
02:13:09.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.131762 23684 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 9 FOLLOWER]: Advancing to term 10
02:13:09.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.136325 23684 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:09.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.137806 23684 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 10 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2:46235)
02:13:09.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.139467 23625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "eccfb2228fdd4bc18088d450500f2742" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "75d6e7b68fdf468cb97ac9fa184bec55"
02:13:09.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.139935 23625 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 8 FOLLOWER]: Advancing to term 10
02:13:09.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.141088 23376 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111)
02:13:09.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.146283 23625 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eccfb2228fdd4bc18088d450500f2742 in term 10.
02:13:09.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.147703 23376 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, eccfb2228fdd4bc18088d450500f2742; no voters: 85d91954824a403e867a4180a90224ea
02:13:09.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.148360 23684 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 10 FOLLOWER]: Leader election won for term 10
02:13:09.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.149845 23684 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 10 LEADER]: Becoming Leader. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Running, Role: LEADER
02:13:09.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.150712 23684 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:09.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.160477 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 reported cstate change: term changed from 9 to 10, leader changed from 85d91954824a403e867a4180a90224ea (127.20.232.3) to eccfb2228fdd4bc18088d450500f2742 (127.20.232.1). New cstate: current_term: 10 leader_uuid: "eccfb2228fdd4bc18088d450500f2742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: UNKNOWN } } }
02:13:09.398 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:09.399 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:13:09.399 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1648), Traces: [0ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [2ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [7ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [9ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [10ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [10ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [28ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [29ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [30ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [31ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [48ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [49ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [51ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [51ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [104ms] refreshing cache from master, [104ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [108ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [109ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [111ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [111ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [524ms] refreshing cache from master, [524ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [529ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [531ms] sending RPC to server 85d91954824a403e867a4180a90224ea, [533ms] received response from server 85d91954824a403e867a4180a90224ea: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [533ms] delaying RPC due to: Network error: Failed to connect to peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111): Connection refused: /127.20.232.3:34111, [1644ms] refreshing cache from master, [1644ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [1648ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@144111092(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@2e6a55b1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421) -> (continuation of Deferred@1995917516 after retry RPC@1447266670) -> (continuation of Deferred@488569335 after retry RPC@1474038723) -> (continuation of Deferred@1268266182 after retry RPC@2083372133), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@70a20d27 -> wakeup thread Time-limited test -> (continuation of Deferred@78259935 after retry RPC@1686231128) -> (continuation of Deferred@1029751001 after retry RPC@1472826421) -> (continuation of Deferred@1995917516 after retry RPC@1447266670) -> (continuation of Deferred@488569335 after retry RPC@1474038723) -> (continuation of Deferred@1268266182 after retry RPC@2083372133))) after lookup
02:13:09.399 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:09.401 [DEBUG - kudu-nio-5] (Connection.java:757) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] Successfully connected to peer
02:13:09.429 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0xe296cb2e, L:/127.0.0.1:45026 - R:/127.20.232.1:38909] using TOKEN/null
02:13:09.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.441762 23625 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 10 FOLLOWER]: Refusing update from remote peer eccfb2228fdd4bc18088d450500f2742: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
02:13:09.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.444036 23684 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.001s
02:13:09.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.445011 23376 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 -> Peer 85d91954824a403e867a4180a90224ea (127.20.232.3:34111): Couldn't send request to peer 85d91954824a403e867a4180a90224ea. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.3:34111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:13:09.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.471513 23702 mvcc.cc:204] Tried to move back new op lower bound from 7305120929533747200 to 7305120928369664000. Current Snapshot: MvccSnapshot[applied={T|T < 7305120929533747200}]
02:13:09.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.482154 23704 mvcc.cc:204] Tried to move back new op lower bound from 7305120929533747200 to 7305120928369664000. Current Snapshot: MvccSnapshot[applied={T|T < 7305120929533747200}]
02:13:09.537 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.3:34111
02:13:09.538 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.539731 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:13:09.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:34111
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37977
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:13:09.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.880496 23711 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:13:09.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.880983 23711 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:13:09.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.881448 23711 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:13:09.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.916676 23711 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:13:09.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.917565 23711 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.3
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.954777 23711 tablet_server_runner.cc:78] Tablet server non-default flags:
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.3:34111
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.3
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37977
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.3
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/logs
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:13:09.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:13:09.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:13:09.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:13:09.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:13:09.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:13:09.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:13:09.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.956063 23711 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:13:09.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:09.957597 23711 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:13:09.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.970587 23717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:11.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.971817 23718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:11.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:09.975080 23720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:11.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:11.123319 23719 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1148 milliseconds
02:13:11.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.123433 23711 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:13:11.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.124547 23711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:13:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.127203 23711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:13:11.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.128774 23711 hybrid_clock.cc:648] HybridClock initialized: now 1783476791128681 us; error 82 us; skew 500 ppm
02:13:11.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.129829 23711 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:13:11.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.137274 23711 webserver.cc:533] Webserver started at http://127.20.232.3:37977/ using document root <none> and password file <none>
02:13:11.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.138530 23711 fs_manager.cc:362] Metadata directory not provided
02:13:11.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.138938 23711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:13:11.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.147249 23711 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.001s
02:13:11.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.152067 23729 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:13:11.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.153236 23711 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
02:13:11.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.153591 23711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:13:11.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85d91954824a403e867a4180a90224ea"
02:13:11.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:27 on dist-test-slave-h94b"
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.155633 23711 fs_report.cc:389] FS layout report
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/wal
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/data
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:13:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:13:11.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.219732 23711 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:13:11.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.221580 23711 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:13:11.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.222126 23711 kserver.cc:163] Server-wide thread pool size limit: 3276
02:13:11.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.225279 23711 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:13:11.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.232004 23736 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:13:11.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.242405 23711 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:13:11.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.242682 23711 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.000s
02:13:11.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.242946 23711 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:13:11.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.248360 23711 ts_tablet_manager.cc:616] Registered 1 tablets
02:13:11.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.248575 23711 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.000s	sys 0.003s
02:13:11.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.249364 23736 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap starting.
02:13:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.333803 23736 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Log is configured to *not* fsync() on all Append() calls
02:13:11.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.432389 23711 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.3:34111
02:13:11.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.432559 23843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.3:34111 every 8 connection(s)
02:13:11.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.435158 23711 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-2/data/info.pb
02:13:11.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.445533 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 23711
02:13:11.446 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:13:11.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.457145 23736 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:13:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.462944 23736 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Bootstrap complete.
02:13:11.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.464834 23736 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent bootstrapping tablet: real 0.216s	user 0.157s	sys 0.040s
02:13:11.493 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:13:11.493 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.20.232.1:38909
02:13:11.494 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:13:11.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.494454 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 23352
02:13:11.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.522351 23736 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:11.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.525607 23736 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d91954824a403e867a4180a90224ea, State: Initialized, Role: FOLLOWER
02:13:11.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.526742 23736 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:11.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.540009 23846 heartbeater.cc:344] Connected to a master server at 127.20.232.62:43871
02:13:11.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.540592 23846 heartbeater.cc:461] Registering TS with master...
02:13:11.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.542049 23846 heartbeater.cc:507] Master 127.20.232.62:43871 requested a full tablet report, sending...
02:13:11.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.545218 23845 heartbeater.cc:344] Connected to a master server at 127.20.232.61:38891
02:13:11.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.545598 23845 heartbeater.cc:461] Registering TS with master...
02:13:11.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.546545 23845 heartbeater.cc:507] Master 127.20.232.61:38891 requested a full tablet report, sending...
02:13:11.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.549389 23844 heartbeater.cc:344] Connected to a master server at 127.20.232.60:33125
02:13:11.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.549739 23844 heartbeater.cc:461] Registering TS with master...
02:13:11.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.550681 23844 heartbeater.cc:507] Master 127.20.232.60:33125 requested a full tablet report, sending...
02:13:11.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.550870 21527 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:13:11.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.553457 23736 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea: Time spent starting tablet: real 0.088s	user 0.033s	sys 0.017s
02:13:11.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.554131 21452 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:13:11.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.554579 21608 ts_manager.cc:194] Re-registered known tserver with Master: 85d91954824a403e867a4180a90224ea (127.20.232.3:34111)
02:13:11.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.557435 21608 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.232.3:46963
02:13:11.560 [DEBUG - kudu-nio-5] (Connection.java:239) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] handling channelInactive
02:13:11.560 [DEBUG - kudu-nio-5] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state READY due to: connection closed
02:13:11.561 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:13:11.562 [DEBUG - Time-limited test] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:11.563 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.563 [DEBUG - kudu-nio-6] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.564 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:11.565368 23844 heartbeater.cc:499] Master 127.20.232.60:33125 was elected leader, sending a full tablet report...
02:13:11.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:11.578 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:13:11.579 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [1ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030309882(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@685b442b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4fbaea3a -> wakeup thread Time-limited test)) after lookup
02:13:11.579 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:11.580 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.580 [DEBUG - kudu-nio-7] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.580 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.599 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:11.599 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:13:11.600 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [1ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [17ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030309882(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@685b442b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4fbaea3a -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793))) after lookup
02:13:11.600 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:11.601 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.601 [DEBUG - kudu-nio-0] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.602 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.638 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:11.638 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:13:11.639 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [1ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [17ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [38ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [40ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [76ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030309882(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@685b442b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4fbaea3a -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884))) after lookup
02:13:11.640 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:11.641 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.641 [DEBUG - kudu-nio-1] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.641 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.738 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:11.738 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:13:11.739 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=177), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [1ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [17ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [38ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [40ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [76ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [77ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [79ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [79ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [172ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [176ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030309882(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@685b442b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4fbaea3a -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469))) after lookup
02:13:11.739 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:11.740 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.740 [DEBUG - kudu-nio-2] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.740 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.799 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:11.799 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:13:11.799 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=237), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [1ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [17ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [38ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [40ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [76ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [77ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [79ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [79ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [172ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [176ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [177ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [178ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [178ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [232ms] refreshing cache from master, [232ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [236ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030309882(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@685b442b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469) -> (continuation of Deferred@1197105006 after retry RPC@28590085), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4fbaea3a -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469) -> (continuation of Deferred@1197105006 after retry RPC@28590085))) after lookup
02:13:11.800 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:11.801 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.801 [DEBUG - kudu-nio-3] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.801 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.978 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:11.979 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235),85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)[L]]}]
02:13:11.979 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=417), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [1ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [17ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [38ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [40ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [76ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [77ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [79ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [79ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [172ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [176ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [177ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [178ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [178ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [232ms] refreshing cache from master, [232ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [236ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [238ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [239ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [239ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [412ms] refreshing cache from master, [413ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [416ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030309882(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@685b442b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469) -> (continuation of Deferred@1197105006 after retry RPC@28590085) -> (continuation of Deferred@430130547 after retry RPC@984910378), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4fbaea3a -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469) -> (continuation of Deferred@1197105006 after retry RPC@28590085) -> (continuation of Deferred@430130547 after retry RPC@984910378))) after lookup
02:13:11.980 [DEBUG - kudu-nio-2] (Connection.java:748) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] connecting to peer
02:13:11.981 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.981 [DEBUG - kudu-nio-4] (Connection.java:709) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] cleaning up while in state CONNECTING due to: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:11.981 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909) for tablet 27a5fe9d83dc490c8a3ea41d6a1fff73: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909
02:13:12.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.614642 23856 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:13:12.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.615042 23856 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:12.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.617740 23856 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:13:12.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:12.621445 23559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:13:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:12.624770 23559 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:13:12.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.630798 23799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea" is_pre_election: true
02:13:12.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.631546 23799 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 9.
02:13:12.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.633001 23559 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:13:12.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.633790 23856 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 10 FOLLOWER]: Leader pre-election won for term 11
02:13:12.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.634088 23856 raft_consensus.cc:493] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 10 FOLLOWER]: Starting leader election (detected failure of leader eccfb2228fdd4bc18088d450500f2742)
02:13:12.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.634379 23856 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 10 FOLLOWER]: Advancing to term 11
02:13:12.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.639633 23856 raft_consensus.cc:515] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:12.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.641631 23856 leader_election.cc:290] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 11 election: Requested vote from peers 85d91954824a403e867a4180a90224ea (127.20.232.3:34111), eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909)
02:13:12.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.642694 23799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27a5fe9d83dc490c8a3ea41d6a1fff73" candidate_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "85d91954824a403e867a4180a90224ea"
02:13:12.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.643270 23799 raft_consensus.cc:3060] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 9 FOLLOWER]: Advancing to term 11
02:13:12.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:12.647142 23559 leader_election.cc:336] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111)
02:13:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.652349 23799 raft_consensus.cc:2468] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75d6e7b68fdf468cb97ac9fa184bec55 in term 11.
02:13:12.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.653563 23559 leader_election.cc:304] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75d6e7b68fdf468cb97ac9fa184bec55, 85d91954824a403e867a4180a90224ea; no voters: eccfb2228fdd4bc18088d450500f2742
02:13:12.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.654407 23856 raft_consensus.cc:2804] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 11 FOLLOWER]: Leader election won for term 11
02:13:12.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.655635 23856 raft_consensus.cc:697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [term 11 LEADER]: Becoming Leader. State: Replica: 75d6e7b68fdf468cb97ac9fa184bec55, State: Running, Role: LEADER
02:13:12.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.656558 23856 consensus_queue.cc:237] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:12.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:12.665971 21608 catalog_manager.cc:5697] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 reported cstate change: term changed from 10 to 11, leader changed from eccfb2228fdd4bc18088d450500f2742 (127.20.232.1) to 75d6e7b68fdf468cb97ac9fa184bec55 (127.20.232.2). New cstate: current_term: 11 leader_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } health_report { overall_health: HEALTHY } } }
02:13:13.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:13.203346 23840 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5060 after lost signal to thread 23712
02:13:13.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:13.268364 23799 raft_consensus.cc:1275] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 85d91954824a403e867a4180a90224ea [term 11 FOLLOWER]: Refusing update from remote peer 75d6e7b68fdf468cb97ac9fa184bec55: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
02:13:13.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:13.270328 23875 consensus_queue.cc:1048] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
02:13:13.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:13.301281 23559 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 -> Peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Couldn't send request to peer eccfb2228fdd4bc18088d450500f2742. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:13:14.079 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 27a5fe9d83dc490c8a3ea41d6a1fff73 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998' with partition [<start>, <end>)
02:13:14.079 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73@[75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)[L],85d91954824a403e867a4180a90224ea(127.20.232.3:34111),eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)]}]
02:13:14.080 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2518), Traces: [0ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [1ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [16ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [17ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [18ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [18ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [37ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [38ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [40ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [40ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [76ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [77ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [79ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [79ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [172ms] refreshing cache from master, [172ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [176ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [177ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [178ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [178ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [232ms] refreshing cache from master, [232ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [236ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [238ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [239ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [239ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [412ms] refreshing cache from master, [413ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [416ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, [418ms] sending RPC to server eccfb2228fdd4bc18088d450500f2742, [419ms] received response from server eccfb2228fdd4bc18088d450500f2742: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [419ms] delaying RPC due to: Network error: Failed to connect to peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909): Connection refused: /127.20.232.1:38909, [2512ms] refreshing cache from master, [2513ms] Sub RPC GetTableLocations: sending RPC to server master-127.20.232.60:33125, [2517ms] Sub RPC GetTableLocations: received response from server master-127.20.232.60:33125: OK, deferred=Deferred@2030309882(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$237/0x00007f9b9c320c88@685b442b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469) -> (continuation of Deferred@1197105006 after retry RPC@28590085) -> (continuation of Deferred@430130547 after retry RPC@984910378) -> (continuation of Deferred@400749761 after retry RPC@161073613), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4fbaea3a -> wakeup thread Time-limited test -> (continuation of Deferred@1455954800 after retry RPC@1312538793) -> (continuation of Deferred@323696188 after retry RPC@65376884) -> (continuation of Deferred@1615796394 after retry RPC@1326858469) -> (continuation of Deferred@1197105006 after retry RPC@28590085) -> (continuation of Deferred@430130547 after retry RPC@984910378) -> (continuation of Deferred@400749761 after retry RPC@161073613))) after lookup
02:13:14.080 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] connecting to peer
02:13:14.081 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] Successfully connected to peer
02:13:14.103 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0xd5f7139f, L:/127.0.0.1:36818 - R:/127.20.232.2:46235] using TOKEN/null
02:13:14.159 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.20.232.1:38909
02:13:14.160 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:14.161697 21408 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:14.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:14.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:13:14.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:14.511045 23896 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:13:14.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:14.511718 23896 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:13:14.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:14.512523 23896 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:13:14.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:14.565841 23896 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:13:14.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:14.566753 23896 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.232.1
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:14.606685 23896 tablet_server_runner.cc:78] Tablet server non-default flags:
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.232.20:42003
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.232.1:38909
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.232.1
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33627
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.232.62:43871,127.20.232.61:38891,127.20.232.60:33125
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.232.1
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/logs
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:13:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
02:13:14.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:13:14.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:43:34 UTC on e7f111948823
02:13:14.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12767
02:13:14.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:13:14.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:14.607966 23896 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:13:14.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:14.609599 23896 file_cache.cc:504] Constructed file cache file cache with capacity 419430
02:13:14.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:14.624046 23903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:15.784435 23559 consensus_peers.cc:597] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P 75d6e7b68fdf468cb97ac9fa184bec55 -> Peer eccfb2228fdd4bc18088d450500f2742 (127.20.232.1:38909): Couldn't send request to peer eccfb2228fdd4bc18088d450500f2742. Status: Network error: Client connection negotiation failed: client connection to 127.20.232.1:38909: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:13:16.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:14.626924 23904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:16.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.238757 23896 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.613s	user 0.545s	sys 0.910s
02:13:16.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.025070 23902 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 23896
02:13:16.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.240034 23905 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1614 milliseconds
02:13:16.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.240062 23896 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.616s	user 0.546s	sys 0.910s
02:13:16.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.241333 23906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:13:16.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.241304 23896 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:13:16.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.245637 23896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:13:16.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.248683 23896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:13:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.250289 23896 hybrid_clock.cc:648] HybridClock initialized: now 1783476796250223 us; error 44 us; skew 500 ppm
02:13:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.251657 23896 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:13:16.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.259130 23896 webserver.cc:533] Webserver started at http://127.20.232.1:33627/ using document root <none> and password file <none>
02:13:16.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.260515 23896 fs_manager.cc:362] Metadata directory not provided
02:13:16.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.260773 23896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:13:16.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.270426 23896 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.002s	sys 0.005s
02:13:16.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.276798 23921 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:13:16.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.278430 23896 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
02:13:16.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.278857 23896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data,/tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:16.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "eccfb2228fdd4bc18088d450500f2742"
02:13:16.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:12:22 on dist-test-slave-h94b"
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.281562 23896 fs_report.cc:389] FS layout report
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/wal
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/data
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:13:16.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:13:16.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.347087 23896 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:13:16.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.349129 23896 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:13:16.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.349757 23896 kserver.cc:163] Server-wide thread pool size limit: 3276
02:13:16.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.353453 23896 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:13:16.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.361949 23928 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:13:16.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.374629 23896 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:13:16.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.375283 23896 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.015s	user 0.002s	sys 0.002s
02:13:16.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.375801 23896 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:13:16.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.383745 23896 ts_tablet_manager.cc:616] Registered 1 tablets
02:13:16.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.384346 23896 ts_tablet_manager.cc:595] Time spent register tablets: real 0.009s	user 0.008s	sys 0.000s
02:13:16.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.384526 23928 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap starting.
02:13:16.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.459573 23928 log.cc:826] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Log is configured to *not* fsync() on all Append() calls
02:13:16.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.599565 23896 rpc_server.cc:307] RPC server started. Bound to: 127.20.232.1:38909
02:13:16.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.599709 24035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.232.1:38909 every 8 connection(s)
02:13:16.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.601887 23928 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:13:16.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.602898 23928 tablet_bootstrap.cc:492] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Bootstrap complete.
02:13:16.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.603102 23896 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskttcaKe/test-tmp/mini-kudu-cluster10059136456063586480/ts-0/data/info.pb
02:13:16.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.604509 23928 ts_tablet_manager.cc:1403] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent bootstrapping tablet: real 0.220s	user 0.181s	sys 0.031s
02:13:16.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.607298 21408 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu as pid 23896
02:13:16.608 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:13:16.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.635272 23928 raft_consensus.cc:359] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:16.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.640045 23928 raft_consensus.cc:740] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccfb2228fdd4bc18088d450500f2742, State: Initialized, Role: FOLLOWER
02:13:16.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.642207 23928 consensus_queue.cc:260] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d91954824a403e867a4180a90224ea" member_type: VOTER last_known_addr { host: "127.20.232.3" port: 34111 } } peers { permanent_uuid: "eccfb2228fdd4bc18088d450500f2742" member_type: VOTER last_known_addr { host: "127.20.232.1" port: 38909 } } peers { permanent_uuid: "75d6e7b68fdf468cb97ac9fa184bec55" member_type: VOTER last_known_addr { host: "127.20.232.2" port: 46235 } }
02:13:16.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.652802 23928 ts_tablet_manager.cc:1434] T 27a5fe9d83dc490c8a3ea41d6a1fff73 P eccfb2228fdd4bc18088d450500f2742: Time spent starting tablet: real 0.048s	user 0.042s	sys 0.000s
02:13:16.659 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:13:16.674 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476747998, tablet=27a5fe9d83dc490c8a3ea41d6a1fff73, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:13:16.680 [DEBUG - main] (Connection.java:575) [peer master-127.20.232.60:33125(127.20.232.60:33125)] disconnecting while in state READY
02:13:16.681 [DEBUG - main] (Connection.java:575) [peer 85d91954824a403e867a4180a90224ea(127.20.232.3:34111)] disconnecting while in state TERMINATED
02:13:16.685 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.20.232.60:33125(127.20.232.60:33125)] handling channelInactive
02:13:16.685 [DEBUG - main] (Connection.java:575) [peer master-127.20.232.62:43871(127.20.232.62:43871)] disconnecting while in state READY
02:13:16.685 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.20.232.60:33125(127.20.232.60:33125)] cleaning up while in state READY due to: connection closed
02:13:16.687 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.20.232.62:43871(127.20.232.62:43871)] handling channelInactive
02:13:16.688 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.20.232.62:43871(127.20.232.62:43871)] cleaning up while in state READY due to: connection closed
02:13:16.689 [DEBUG - main] (Connection.java:575) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] disconnecting while in state READY
02:13:16.690 [DEBUG - main] (Connection.java:575) [peer eccfb2228fdd4bc18088d450500f2742(127.20.232.1:38909)] disconnecting while in state TERMINATED
02:13:16.691 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] handling channelInactive
02:13:16.691 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 75d6e7b68fdf468cb97ac9fa184bec55(127.20.232.2:46235)] cleaning up while in state READY due to: connection closed
02:13:16.692 [DEBUG - main] (Connection.java:575) [peer master-127.20.232.61:38891(127.20.232.61:38891)] disconnecting while in state READY
02:13:16.693 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.20.232.61:38891(127.20.232.61:38891)] handling channelInactive
02:13:16.693 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.20.232.61:38891(127.20.232.61:38891)] cleaning up while in state READY due to: connection closed
02:13:16.695 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
02:13:16.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.702771 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 23896
02:13:16.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.733078 24045 negotiation.cc:336] Failed RPC negotiation. Trace:
02:13:16.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.682220 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.232.1:49059 (local address 127.20.232.62:43871)
02:13:16.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.708789 (+ 26569us) server_negotiation.cc:207] Beginning negotiation
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.708820 (+    31us) server_negotiation.cc:400] Waiting for connection header
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.731674 (+ 22854us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.232.1:49059: BlockingRecv error: recv got EOF from 127.20.232.1:49059 (error 108)
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":25866,"thread_start_us":25479,"threads_started":1}
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.733582 24041 negotiation.cc:336] Failed RPC negotiation. Trace:
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.617151 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.232.1:53073 (local address 127.20.232.60:33125)
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.644819 (+ 27668us) server_negotiation.cc:207] Beginning negotiation
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.644848 (+    29us) server_negotiation.cc:400] Waiting for connection header
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.661215 (+ 16367us) server_negotiation.cc:408] Connection header received
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.661398 (+   183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:13:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.661426 (+    28us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.661666 (+   240us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.661851 (+   185us) server_negotiation.cc:227] Negotiated authn=SASL
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.690655 (+ 28804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.697888 (+  7233us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.704798 (+  6910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.705443 (+   645us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.731889 (+ 26446us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.232.1:53073: BlockingRecv error: recv got EOF from 127.20.232.1:53073 (error 108)
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":20115,"thread_start_us":23020,"threads_started":1}
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.733527 24043 negotiation.cc:336] Failed RPC negotiation. Trace:
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.650616 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.20.232.1:56517 (local address 127.20.232.61:38891)
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.666500 (+ 15884us) server_negotiation.cc:207] Beginning negotiation
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.666533 (+    33us) server_negotiation.cc:400] Waiting for connection header
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.666596 (+    63us) server_negotiation.cc:408] Connection header received
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.666818 (+   222us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.666846 (+    28us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.667085 (+   239us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.667295 (+   210us) server_negotiation.cc:227] Negotiated authn=SASL
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.705371 (+ 38076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.707204 (+  1833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:13:16.731783 (+ 24579us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.232.1:56517: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15120,"thread_start_us":16905,"threads_started":1}
02:13:16.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.735067 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 23536
02:13:16.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.765019 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 23711
02:13:16.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.792243 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 21420
02:13:16.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.820796 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 21490
02:13:16.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:13:16.851046 21594 connection.cc:573] client connection to 127.20.232.61:38891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:13:16.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:13:16.851444 21408 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskttcaKe/build/tsan/bin/kudu with pid 21565
02:13:17.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:13:17Z chronyd exiting

Time: 66.532

OK (1 test)

