JUnit version 4.13.2
.08:20:10.762 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1786522810752
08:20:10.834 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
08:20:10.907 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
08:20:10.922 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskjuwuwQ/test-tmp
08:20:10.929 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
08:20:10.932 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskjuwuwQ/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
08:20:11.155 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

08:20:11.343 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:20:11.344 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

08:20:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.351298  3122 mini_oidc.cc:150] Starting JWKS server
08:20:11.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.356380  3122 webserver.cc:533] Webserver started at http://127.0.0.1:34769/ using document root <none> and password file <none>
08:20:11.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.356916  3122 mini_oidc.cc:186] Starting OIDC Discovery server
08:20:11.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.359812  3122 webserver.cc:533] Webserver started at http://127.0.0.1:46567/ using document root <none> and password file <none>
08:20:11.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:20:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
08:20:11.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:20:11Z Disabled control of system clock
08:20:11.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.435814  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:11.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:11.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/wal
08:20:11.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data
08:20:11.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:11.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:11.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:11.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:11.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:11.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/logs
08:20:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data/info.pb
08:20:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:11.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:11.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:11.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:11.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:11.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:11.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:11.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:20:11.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:11.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:20:11.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:20:11.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.190:33597
08:20:11.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.190
08:20:11.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:11.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:11.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:11.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:11.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:11.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:20:11.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:11.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:20:11.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34769/jwks.json with env {}
08:20:11.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.786971  3149 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:11.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.787708  3149 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:11.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.788272  3149 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:11.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.826299  3149 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:20:11.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.826633  3149 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:11.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.826838  3149 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:20:11.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.827085  3149 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:20:11.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.827283  3149 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:20:11.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.827426  3149 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34769/jwks.json
08:20:11.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.869836  3149 master_runner.cc:387] Master server non-default flags:
08:20:11.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/logs
08:20:11.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:11.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:11.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:11.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:11.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:11.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data
08:20:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/wal
08:20:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:20:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:11.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:20:11.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.190:33597
08:20:11.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:20:11.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:11.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:20:11.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34769/jwks.json
08:20:11.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:11.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:11.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data/info.pb
08:20:11.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.190
08:20:11.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:11.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:11.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:11.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:11.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:20:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:11.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:11.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:11.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:11.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:11.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.871492  3149 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:11.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:11.873458  3149 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:11.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.888218  3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:13.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:11.889055  3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:13.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.252239  3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:13.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.254855  3165 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1360 milliseconds
08:20:13.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.260010  3149 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.373s	user 0.434s	sys 0.933s
08:20:13.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.260340  3149 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.373s	user 0.434s	sys 0.933s
08:20:13.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.260700  3149 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:13.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.262253  3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:13.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.265827  3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:13.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.269527  3149 hybrid_clock.cc:648] HybridClock initialized: now 1786522813269448 us; error 144 us; skew 500 ppm
08:20:13.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.270982  3149 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:13.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.280913  3149 webserver.cc:533] Webserver started at http://127.3.12.190:45813/ using document root <none> and password file <none>
08:20:13.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.282265  3149 fs_manager.cc:362] Metadata directory not provided
08:20:13.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.282521  3149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:13.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.283180  3149 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:20:13.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.288303  3149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data/instance:
08:20:13.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5"
08:20:13.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:13 on dist-test-slave-t7g5"
08:20:13.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.290064  3149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/wal/instance:
08:20:13.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5"
08:20:13.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:13 on dist-test-slave-t7g5"
08:20:13.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.301684  3149 fs_manager.cc:696] Time spent creating directory manager: real 0.011s	user 0.005s	sys 0.005s
08:20:13.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.308229  3178 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:13.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.309834  3149 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
08:20:13.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.310168  3149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/wal
08:20:13.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5"
08:20:13.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:13 on dist-test-slave-t7g5"
08:20:13.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.310531  3149 fs_report.cc:389] FS layout report
08:20:13.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:13.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/wal
08:20:13.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/wal
08:20:13.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data/data
08:20:13.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:13.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:13.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:13.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:13.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:13.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:13.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:13.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:13.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:13.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:13.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:13.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.387692  3149 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:13.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.389993  3149 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:13.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.390632  3149 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:13.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.482934  3149 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.190:33597
08:20:13.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.483039  3260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.190:33597 every 8 connection(s)
08:20:13.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.486416  3149 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/data/info.pb
08:20:13.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.493570  3261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:20:13.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.494881  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 3149
08:20:13.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.495481  3122 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-0/wal/instance
08:20:13.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.500816  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:13.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:13.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/wal
08:20:13.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data
08:20:13.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:13.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:13.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:13.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:13.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:13.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:13.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/logs
08:20:13.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data/info.pb
08:20:13.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:13.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:20:13.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:13.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:20:13.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:20:13.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.189:34545
08:20:13.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.189
08:20:13.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:13.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:13.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:13.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:13.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:13.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:20:13.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:13.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:20:13.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34769/jwks.json with env {}
08:20:13.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.509954  3261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
08:20:13.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.550338  3261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
08:20:13.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.553864  3184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:13.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.560320  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.189:34545: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:13.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.609974  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } attempt: 1
08:20:13.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.616073  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.189:34545: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:13.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.684576  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } attempt: 2
08:20:13.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.692831  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.189:34545: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:13.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.784343  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } attempt: 3
08:20:13.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.789796  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.189:34545: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:13.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.850356  3266 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:13.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.851094  3266 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:13.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.852012  3266 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:13.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.887816  3266 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:20:13.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.888191  3266 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:13.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.888530  3266 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:20:13.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.888885  3266 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:20:13.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.889250  3266 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:20:13.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.889549  3266 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34769/jwks.json
08:20:13.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.933295  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } attempt: 4
08:20:13.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.934392  3266 master_runner.cc:387] Master server non-default flags:
08:20:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/logs
08:20:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:13.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:13.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:13.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:13.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data
08:20:13.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/wal
08:20:13.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:20:13.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:13.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:13.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:13.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:20:13.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.189:34545
08:20:13.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:20:13.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:13.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:20:13.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34769/jwks.json
08:20:13.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:13.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:13.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data/info.pb
08:20:13.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.189
08:20:13.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:13.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:13.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:13.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:13.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:13.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:13.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:13.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:20:13.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:13.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:13.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:13.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:13.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:13.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:13.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.936403  3266 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:13.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.939280  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.189:34545: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:13.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:13.939440  3266 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:13.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.952752  3287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:14.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:14.242553  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } attempt: 5
08:20:14.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:14.250905  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.189:34545: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:14.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:14.799305  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } attempt: 6
08:20:14.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:14.829404  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.189:34545: Network error: Client connection negotiation failed: client connection to 127.3.12.189:34545: connect: Connection refused (error 111)
08:20:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.156872  3256 debug-util.cc:398] Leaking SignalData structure 0x7b0800070060 after lost signal to thread 3155
08:20:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.159236  3256 debug-util.cc:398] Leaking SignalData structure 0x7b08000707a0 after lost signal to thread 3260
08:20:15.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.358733  3284 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee60 after lost signal to thread 3266
08:20:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.366504  3284 kernel_stack_watchdog.cc:198] Thread 3266 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
08:20:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:20:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:20:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:20:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:20:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.367511  3266 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.414s	user 0.542s	sys 0.863s
08:20:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.367931  3266 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.415s	user 0.542s	sys 0.863s
08:20:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:13.954909  3289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:15.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.368868  3290 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1412 milliseconds
08:20:15.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.370333  3291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:15.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.371153  3266 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:15.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.375047  3266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:15.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.378736  3266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:15.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.380353  3266 hybrid_clock.cc:648] HybridClock initialized: now 1786522815380267 us; error 124 us; skew 500 ppm
08:20:15.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.382174  3266 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:15.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.390944  3266 webserver.cc:533] Webserver started at http://127.3.12.189:39169/ using document root <none> and password file <none>
08:20:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.392313  3266 fs_manager.cc:362] Metadata directory not provided
08:20:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.392606  3266 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:15.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.393217  3266 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:20:15.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.399941  3266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data/instance:
08:20:15.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "702a183e24184c569c6bb62bc271dfa2"
08:20:15.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:15 on dist-test-slave-t7g5"
08:20:15.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.401692  3266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/wal/instance:
08:20:15.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "702a183e24184c569c6bb62bc271dfa2"
08:20:15.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:15 on dist-test-slave-t7g5"
08:20:15.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.412492  3266 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.005s	sys 0.005s
08:20:15.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.420987  3309 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:15.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.422415  3266 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
08:20:15.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.422778  3266 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/wal
08:20:15.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "702a183e24184c569c6bb62bc271dfa2"
08:20:15.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:15 on dist-test-slave-t7g5"
08:20:15.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.423213  3266 fs_report.cc:389] FS layout report
08:20:15.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:15.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/wal
08:20:15.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/wal
08:20:15.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data/data
08:20:15.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:15.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:15.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:15.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:15.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:15.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.499713  3266 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:15.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.501300  3266 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:15.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.501789  3266 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:15.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.593204  3266 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.189:34545
08:20:15.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.593282  3396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.189:34545 every 8 connection(s)
08:20:15.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.596722  3266 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/data/info.pb
08:20:15.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.605301  3397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:20:15.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.606375  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 3266
08:20:15.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.606868  3122 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-1/wal/instance
08:20:15.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.609783  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:15.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:15.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/wal
08:20:15.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data
08:20:15.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:15.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:15.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:15.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:15.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:15.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:15.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/logs
08:20:15.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data/info.pb
08:20:15.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:15.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:20:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:20:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:20:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.188:42903
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.188
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:15.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:20:15.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34769/jwks.json with env {}
08:20:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.623749  3397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
08:20:15.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.658615  3397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
08:20:15.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.678011  3397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
08:20:15.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.681260  3313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:15.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.686769  3397 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:15.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.736423  3397 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 1
08:20:15.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.741972  3397 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:15.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.810446  3397 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 2
08:20:15.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.815976  3397 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:15.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.890069  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } attempt: 7
08:20:15.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.905259  3261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
08:20:15.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.907416  3397 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 3
08:20:15.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.913064  3397 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:15.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.917272  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:15.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:15.975761  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 1
08:20:15.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:15.981078  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:16.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.004813  3406 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:16.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.005515  3406 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:16.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.006224  3406 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:16.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.041847  3406 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:20:16.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.042215  3406 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:16.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.042558  3406 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:20:16.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.042979  3406 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:20:16.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.043298  3406 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:20:16.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.043551  3406 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34769/jwks.json
08:20:16.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.056582  3397 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 4
08:20:16.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.062546  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 2
08:20:16.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.063805  3397 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:16.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.068425  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:16.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.106240  3406 master_runner.cc:387] Master server non-default flags:
08:20:16.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/logs
08:20:16.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:16.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:16.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:16.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:16.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/wal
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.188:42903
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34769/jwks.json
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data/info.pb
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.188
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.108275  3406 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.110689  3406 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:16.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.125240  3425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:16.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.153859  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 3
08:20:16.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.160982  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:16.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.301501  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 4
08:20:16.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.309718  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:16.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.363257  3397 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 5
08:20:16.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.371584  3397 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:16.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.593217  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 5
08:20:16.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.600685  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:16.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:16.919087  3397 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 6
08:20:16.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.938812  3397 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:17.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.153200  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 6
08:20:17.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.180074  3261 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.12.188:42903: Network error: Client connection negotiation failed: client connection to 127.3.12.188:42903: connect: Connection refused (error 111)
08:20:17.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.527944  3424 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee60 after lost signal to thread 3406
08:20:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.615134  3424 kernel_stack_watchdog.cc:198] Thread 3406 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
08:20:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:20:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:20:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:20:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:20:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.616290  3427 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1489 milliseconds
08:20:17.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.616855  3406 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.491s	user 0.002s	sys 0.000s
08:20:17.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:16.126094  3426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:17.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.617228  3406 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.492s	user 0.003s	sys 0.000s
08:20:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.619164  3406 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.619158  3428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:17.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.623463  3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:17.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.627296  3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.628875  3406 hybrid_clock.cc:648] HybridClock initialized: now 1786522817628795 us; error 83 us; skew 500 ppm
08:20:17.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.629959  3406 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:17.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.638197  3406 webserver.cc:533] Webserver started at http://127.3.12.188:37911/ using document root <none> and password file <none>
08:20:17.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.639643  3406 fs_manager.cc:362] Metadata directory not provided
08:20:17.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.640085  3406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:17.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.640588  3406 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:20:17.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.645635  3406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data/instance:
08:20:17.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f"
08:20:17.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:17 on dist-test-slave-t7g5"
08:20:17.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.646953  3406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/wal/instance:
08:20:17.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f"
08:20:17.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:17 on dist-test-slave-t7g5"
08:20:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.656431  3406 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.009s	sys 0.000s
08:20:17.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.663868  3450 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.665596  3406 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.004s
08:20:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.666015  3406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/wal
08:20:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f"
08:20:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:17 on dist-test-slave-t7g5"
08:20:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.666364  3406 fs_report.cc:389] FS layout report
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/wal
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/wal
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data/data
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:17.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:17.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:17.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:17.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.732211  3406 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:17.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.733860  3406 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:17.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.734833  3406 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:17.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.819425  3406 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.188:42903
08:20:17.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.819693  3535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.188:42903 every 8 connection(s)
08:20:17.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.822775  3406 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/data/info.pb
08:20:17.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.823719  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 3406
08:20:17.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.824218  3122 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/master-2/wal/instance
08:20:17.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.830950  3537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:20:17.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.847846  3537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
08:20:17.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.877753  3537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
08:20:17.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.889962  3537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
08:20:17.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.909878  3537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f: Bootstrap starting.
08:20:17.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.918427  3537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f: Neither blocks nor log segments found. Creating new log.
08:20:17.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.921083  3537 log.cc:826] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f: Log is configured to *not* fsync() on all Append() calls
08:20:17.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.928198  3537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f: No bootstrap required, opened a new log
08:20:17.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.950960  3537 raft_consensus.cc:359] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:17.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.951926  3537 raft_consensus.cc:385] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:20:17.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.952242  3537 raft_consensus.cc:740] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ddd4d3d4cb4b4e9a866ab9e0e7493f, State: Initialized, Role: FOLLOWER
08:20:17.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.953275  3537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:17.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.956497  3559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.957301  3559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: This master's current role is: FOLLOWER
08:20:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.958524  3537 sys_catalog.cc:565] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: configured and running, proceeding with master startup.
08:20:17.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.990564  3579 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:20:17.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.991218  3457 tablet.cc:2406] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:20:17.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:17.991567  3579 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:20:17.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.996943  3559 raft_consensus.cc:493] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:20:17.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.997804  3559 raft_consensus.cc:515] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:18.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.999368  3397 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 7
08:20:18.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.001708  3559 leader_election.cc:290] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e58d7e8fef1c40e2b8f30ecf403dc7d5 (127.3.12.190:33597), 702a183e24184c569c6bb62bc271dfa2 (127.3.12.189:34545)
08:20:18.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:17.985357  3406 master_runner.cc:187] Error getting master registration for 127.3.12.190:33597: OK, instance_id { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" instance_seqno: 1786522813458550 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:20:18.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.026182  3406 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
08:20:18.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.031162  3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: Bootstrap starting.
08:20:18.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.038829  3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: Neither blocks nor log segments found. Creating new log.
08:20:18.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.041846  3397 log.cc:826] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: Log is configured to *not* fsync() on all Append() calls
08:20:18.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.056391  3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: No bootstrap required, opened a new log
08:20:18.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.080932  3397 raft_consensus.cc:359] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.082070  3397 raft_consensus.cc:385] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:20:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.082600  3397 raft_consensus.cc:740] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702a183e24184c569c6bb62bc271dfa2, State: Initialized, Role: FOLLOWER
08:20:18.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.084043  3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:18.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.089368  3397 sys_catalog.cc:565] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [sys.catalog]: configured and running, proceeding with master startup.
08:20:18.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.089534  3356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702a183e24184c569c6bb62bc271dfa2" is_pre_election: true
08:20:18.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.090737  3356 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ddd4d3d4cb4b4e9a866ab9e0e7493f in term 0.
08:20:18.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.092767  3455 leader_election.cc:304] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ddd4d3d4cb4b4e9a866ab9e0e7493f, 702a183e24184c569c6bb62bc271dfa2; no voters: 
08:20:18.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.089107  3584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.094033  3559 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Leader pre-election won for term 1
08:20:18.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.093772  3584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [sys.catalog]: This master's current role is: FOLLOWER
08:20:18.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.094461  3559 raft_consensus.cc:493] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:20:18.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.094830  3559 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 0 FOLLOWER]: Advancing to term 1
08:20:18.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.102921  3559 raft_consensus.cc:515] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:18.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.113126  3356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702a183e24184c569c6bb62bc271dfa2"
08:20:18.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.113914  3356 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [term 0 FOLLOWER]: Advancing to term 1
08:20:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.122516  3356 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ddd4d3d4cb4b4e9a866ab9e0e7493f in term 1.
08:20:18.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.124357  3455 leader_election.cc:304] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ddd4d3d4cb4b4e9a866ab9e0e7493f, 702a183e24184c569c6bb62bc271dfa2; no voters: 
08:20:18.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.127668  3559 leader_election.cc:290] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [CANDIDATE]: Term 1 election: Requested vote from peers e58d7e8fef1c40e2b8f30ecf403dc7d5 (127.3.12.190:33597), 702a183e24184c569c6bb62bc271dfa2 (127.3.12.189:34545)
08:20:18.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.128649  3587 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 1 FOLLOWER]: Leader election won for term 1
08:20:18.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.130457  3587 raft_consensus.cc:697] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [term 1 LEADER]: Becoming Leader. State: Replica: 34ddd4d3d4cb4b4e9a866ab9e0e7493f, State: Running, Role: LEADER
08:20:18.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.131784  3587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:18.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.147913  3559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ddd4d3d4cb4b4e9a866ab9e0e7493f. Latest consensus state: current_term: 1 leader_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.149864  3559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: This master's current role is: LEADER
08:20:18.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.160432  3597 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
08:20:18.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.171496  3604 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
08:20:18.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.174388  3597 catalog_manager.cc:1520] Initializing Kudu cluster ID...
08:20:18.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.210034  3356 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [term 1 FOLLOWER]: Refusing update from remote peer 34ddd4d3d4cb4b4e9a866ab9e0e7493f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:20:18.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.212231  3587 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [LEADER]: Connected to new peer: Peer: permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:20:18.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.213685  3261 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } attempt: 7
08:20:18.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.185268  3266 master_runner.cc:187] Error getting master registration for 127.3.12.190:33597: OK, instance_id { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" instance_seqno: 1786522813458550 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:20:18.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.231657  3266 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
08:20:18.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.251963  3584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ddd4d3d4cb4b4e9a866ab9e0e7493f. Latest consensus state: current_term: 1 leader_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.252754  3584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [sys.catalog]: This master's current role is: FOLLOWER
08:20:18.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.256798  3606 mvcc.cc:204] Tried to move back new op lower bound from 7317597463360147456 to 7317597463081197568. Current Snapshot: MvccSnapshot[applied={T|T < 7317597463360147456}]
08:20:18.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.260468  3261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: Bootstrap starting.
08:20:18.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:18.262360  3318 tablet.cc:2406] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:20:18.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.272362  3559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.273229  3559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: This master's current role is: LEADER
08:20:18.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.274786  3261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: Neither blocks nor log segments found. Creating new log.
08:20:18.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.277832  3597 catalog_manager.cc:1383] Generated new cluster ID: 1a904f64cd884037b5be9de0ed645059
08:20:18.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.278172  3597 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
08:20:18.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.290861  3261 log.cc:826] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: Log is configured to *not* fsync() on all Append() calls
08:20:18.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.292906  3584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.293638  3584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2 [sys.catalog]: This master's current role is: FOLLOWER
08:20:18.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.309697  3261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: No bootstrap required, opened a new log
08:20:18.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.332264  3597 catalog_manager.cc:1406] Generated new certificate authority record
08:20:18.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.335238  3597 catalog_manager.cc:1540] Loading token signing keys...
08:20:18.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.341208  3261 raft_consensus.cc:359] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:18.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.342690  3261 raft_consensus.cc:385] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:20:18.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.343025  3261 raft_consensus.cc:740] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58d7e8fef1c40e2b8f30ecf403dc7d5, State: Initialized, Role: FOLLOWER
08:20:18.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.344014  3261 consensus_queue.cc:260] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } }
08:20:18.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.369303  3597 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f: Generated new TSK 0
08:20:18.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.371352  3261 sys_catalog.cc:565] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [sys.catalog]: configured and running, proceeding with master startup.
08:20:18.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.371809  3597 catalog_manager.cc:1550] Initializing in-progress tserver states...
08:20:18.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.372169  3219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5"
08:20:18.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.371471  3220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" is_pre_election: true
08:20:18.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.376356  3219 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [term 0 FOLLOWER]: Advancing to term 1
08:20:18.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.382488  3616 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.383765  3616 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [sys.catalog]: This master's current role is: FOLLOWER
08:20:18.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.383685  3219 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ddd4d3d4cb4b4e9a866ab9e0e7493f in term 1.
08:20:18.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.393857  3217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [term 1 FOLLOWER]: Refusing update from remote peer 34ddd4d3d4cb4b4e9a866ab9e0e7493f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:20:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.400823  3587 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:20:18.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:18.403867  3186 tablet.cc:2406] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:20:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.406958  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:20:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:0
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:18.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:18.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.499686  3638 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.504439  3616 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ddd4d3d4cb4b4e9a866ab9e0e7493f. Latest consensus state: current_term: 1 leader_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.506649  3638 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [sys.catalog]: This master's current role is: FOLLOWER
08:20:18.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.506942  3616 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5 [sys.catalog]: This master's current role is: FOLLOWER
08:20:18.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.531093  3641 catalog_manager.cc:1295] Loaded cluster ID: 1a904f64cd884037b5be9de0ed645059
08:20:18.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.533792  3641 catalog_manager.cc:1588] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: loading cluster ID for follower catalog manager: success
08:20:18.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.551023  3641 catalog_manager.cc:1610] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: acquiring CA information for follower catalog manager: success
08:20:18.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.557716  3641 catalog_manager.cc:1638] T 00000000000000000000000000000000 P e58d7e8fef1c40e2b8f30ecf403dc7d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:20:18.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.856341  3587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58d7e8fef1c40e2b8f30ecf403dc7d5" member_type: VOTER last_known_addr { host: "127.3.12.190" port: 33597 } } peers { permanent_uuid: "702a183e24184c569c6bb62bc271dfa2" member_type: VOTER last_known_addr { host: "127.3.12.189" port: 34545 } } peers { permanent_uuid: "34ddd4d3d4cb4b4e9a866ab9e0e7493f" member_type: VOTER last_known_addr { host: "127.3.12.188" port: 42903 } } }
08:20:18.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:18.857126  3587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ddd4d3d4cb4b4e9a866ab9e0e7493f [sys.catalog]: This master's current role is: LEADER
08:20:18.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:18.990773  3633 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:18.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:18.991515  3633 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:18.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:18.992231  3633 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:19.057552  3633 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:19.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:19.059376  3633 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.129
08:20:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:19.129617  3633 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:20:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:0
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:20:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:19.131558  3633 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:19.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:19.133994  3633 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:19.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:19.149690  3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:19.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:19.210837  3604 catalog_manager.cc:1295] Loaded cluster ID: 1a904f64cd884037b5be9de0ed645059
08:20:19.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:19.217816  3604 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: loading cluster ID for follower catalog manager: success
08:20:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:19.242551  3604 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: acquiring CA information for follower catalog manager: success
08:20:19.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:19.265990  3604 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 702a183e24184c569c6bb62bc271dfa2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:20:20.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:20.551843  3657 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee40 after lost signal to thread 3633
08:20:20.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:20.837786  3657 kernel_stack_watchdog.cc:198] Thread 3633 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
08:20:20.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:20:20.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:20:20.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:20:20.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:20:20.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:19.151312  3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:20.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:20.839129  3633 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.690s	user 0.558s	sys 0.804s
08:20:20.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:20.839846  3633 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.691s	user 0.558s	sys 0.804s
08:20:20.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.857590  3633 server_base.cc:1061] running on GCE node
08:20:20.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:20.859515  3672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:20.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.861423  3633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:20.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.868073  3633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:20.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.869943  3633 hybrid_clock.cc:648] HybridClock initialized: now 1786522820869893 us; error 60 us; skew 500 ppm
08:20:20.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.871476  3633 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:20.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.891639  3633 webserver.cc:533] Webserver started at http://127.3.12.129:38639/ using document root <none> and password file <none>
08:20:20.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.893564  3633 fs_manager.cc:362] Metadata directory not provided
08:20:20.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.894243  3633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:20.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.895475  3633 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:20:20.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.903474  3633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/instance:
08:20:20.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:20.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:20 on dist-test-slave-t7g5"
08:20:20.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.906327  3633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal/instance:
08:20:20.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:20.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:20 on dist-test-slave-t7g5"
08:20:20.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.923280  3633 fs_manager.cc:696] Time spent creating directory manager: real 0.016s	user 0.013s	sys 0.001s
08:20:20.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.937523  3681 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.939574  3633 fs_manager.cc:730] Time spent opening block manager: real 0.011s	user 0.010s	sys 0.001s
08:20:20.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.940328  3633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:20.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:20.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:20 on dist-test-slave-t7g5"
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:20.941282  3633 fs_report.cc:389] FS layout report
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/data
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:21.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.033895  3633 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:21.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.036764  3633 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:21.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.037799  3633 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.047595  3633 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:21.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.061579  3633 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:20:21.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.062347  3633 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.001s	sys 0.000s
08:20:21.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.062986  3633 ts_tablet_manager.cc:616] Registered 0 tablets
08:20:21.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.063474  3633 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:20:21.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.928757  3885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.129:46667 every 8 connection(s)
08:20:21.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.930140  3633 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.129:46667
08:20:21.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.934564  3633 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:21.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.937533  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 3633
08:20:21.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.938553  3122 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal/instance
08:20:21.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:21.954627  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:21.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:21.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:0
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:21.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:21.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:21.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:22.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.024636  3891 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:22.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.025239  3891 heartbeater.cc:461] Registering TS with master...
08:20:22.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.030504  3891 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:22.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.035022  3478 ts_manager.cc:194] Registered new tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.038298  3478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.129:58109
08:20:22.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.044858  3895 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:22.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.045274  3895 heartbeater.cc:461] Registering TS with master...
08:20:22.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.046316  3895 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:22.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.048910  3900 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:22.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.049252  3900 heartbeater.cc:461] Registering TS with master...
08:20:22.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.050230  3900 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:22.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.050704  3340 ts_manager.cc:194] Registered new tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:22.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.053853  3204 ts_manager.cc:194] Registered new tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:22.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:22.400336  3901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:22.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:22.400935  3901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:22.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:22.401625  3901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:22.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:22.441442  3901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:22.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:22.442503  3901 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.130
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.487733  3901 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:20:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:0
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:20:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:22.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.489581  3901 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:22.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:22.491842  3901 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:22.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:22.507393  3919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:23.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.059648  3891 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:23.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:22.510141  3920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:23.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:23.750751  3901 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.244s	user 0.410s	sys 0.820s
08:20:23.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:23.751180  3901 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.245s	user 0.410s	sys 0.820s
08:20:23.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.753288  3901 server_base.cc:1061] running on GCE node
08:20:23.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:23.754725  3929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:23.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.756455  3901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:23.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.760228  3901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:23.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.762069  3901 hybrid_clock.cc:648] HybridClock initialized: now 1786522823762018 us; error 69 us; skew 500 ppm
08:20:23.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.763417  3901 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:23.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.773237  3901 webserver.cc:533] Webserver started at http://127.3.12.130:39405/ using document root <none> and password file <none>
08:20:23.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.774685  3901 fs_manager.cc:362] Metadata directory not provided
08:20:23.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.775015  3901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:23.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.775648  3901 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:20:23.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.782814  3901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/instance:
08:20:23.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:20:23.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:23 on dist-test-slave-t7g5"
08:20:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.784500  3901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal/instance:
08:20:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:20:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:23 on dist-test-slave-t7g5"
08:20:23.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.796957  3901 fs_manager.cc:696] Time spent creating directory manager: real 0.011s	user 0.005s	sys 0.009s
08:20:23.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.803709  3939 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:23.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.805032  3901 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
08:20:23.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.805346  3901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:23.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:20:23.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:23 on dist-test-slave-t7g5"
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.805709  3901 fs_report.cc:389] FS layout report
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/data
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:23.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:23.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.896948  3901 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:23.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.899080  3901 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:23.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.899623  3901 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:23.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.903319  3901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:23.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.908807  3901 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:20:23.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.909178  3901 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:20:23.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.909832  3901 ts_tablet_manager.cc:616] Registered 0 tablets
08:20:23.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:23.910020  3901 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:20:24.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.076889  3901 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.130:37191
08:20:24.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.077059  4147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.130:37191 every 8 connection(s)
08:20:24.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.080001  3901 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:24.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.084887  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 3901
08:20:24.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.085425  3122 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal/instance
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.111788  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:24.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:0
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:24.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:24.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:24.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:24.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.161813  4148 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:24.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.162535  4148 heartbeater.cc:461] Registering TS with master...
08:20:24.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.163102  4149 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:24.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.163540  4149 heartbeater.cc:461] Registering TS with master...
08:20:24.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.164482  4148 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:24.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.165109  4149 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:24.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.169049  3340 ts_manager.cc:194] Registered new tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.169283  3478 ts_manager.cc:194] Registered new tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.171263  3478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.130:42459
08:20:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.172667  4151 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.172982  4151 heartbeater.cc:461] Registering TS with master...
08:20:24.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.173995  4151 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:24.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.177266  3204 ts_manager.cc:194] Registered new tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:24.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:24.644573  4170 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:24.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:24.645534  4170 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:24.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:24.646145  4170 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:24.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:24.684444  4170 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:24.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:24.685416  4170 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.131
08:20:24.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.729243  4170 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:0
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:24.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.731253  4170 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:24.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:24.733192  4170 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:24.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:24.747340  4188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:25.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:25.176750  4148 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:26.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:26.155977  4187 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee40 after lost signal to thread 4170
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:26.261206  4187 kernel_stack_watchdog.cc:198] Thread 4170 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 405ms:
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:26.261513  4170 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.514s	user 0.600s	sys 0.887s
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:26.261871  4170 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.514s	user 0.600s	sys 0.887s
08:20:26.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:24.748001  4189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:26.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:26.263319  4190 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1514 milliseconds
08:20:26.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:26.263962  4191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:26.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.263901  4170 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:26.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.267588  4170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:26.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.269974  4170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:26.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.271379  4170 hybrid_clock.cc:648] HybridClock initialized: now 1786522826271338 us; error 73 us; skew 500 ppm
08:20:26.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.272318  4170 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:26.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.280460  4170 webserver.cc:533] Webserver started at http://127.3.12.131:42909/ using document root <none> and password file <none>
08:20:26.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.281705  4170 fs_manager.cc:362] Metadata directory not provided
08:20:26.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.281965  4170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:26.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.282503  4170 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:20:26.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.288146  4170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/instance:
08:20:26.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:20:26.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:26 on dist-test-slave-t7g5"
08:20:26.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.289433  4170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal/instance:
08:20:26.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:20:26.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:26 on dist-test-slave-t7g5"
08:20:26.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.301638  4170 fs_manager.cc:696] Time spent creating directory manager: real 0.012s	user 0.007s	sys 0.003s
08:20:26.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.308992  4206 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:26.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.310379  4170 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.310724  4170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:26 on dist-test-slave-t7g5"
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.311183  4170 fs_report.cc:389] FS layout report
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:26.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/data
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:26.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:26.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.356892  4170 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:26.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.358548  4170 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:26.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.358999  4170 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:26.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.362040  4170 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:26.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.367720  4170 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:20:26.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.368001  4170 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:20:26.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.368216  4170 ts_tablet_manager.cc:616] Registered 0 tablets
08:20:26.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.368345  4170 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:20:26.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.531741  4170 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.131:42197
08:20:26.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.531860  4407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.131:42197 every 8 connection(s)
08:20:26.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.534787  4170 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.540411  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 4170
08:20:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.540952  3122 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal/instance
08:20:26.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.598387  4410 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:26.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.602088  4410 heartbeater.cc:461] Registering TS with master...
08:20:26.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.603624  4410 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:26.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.607372  3340 ts_manager.cc:194] Registered new tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:20:26.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.611572  4414 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:26.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.611969  4414 heartbeater.cc:461] Registering TS with master...
08:20:26.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.612972  4409 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:26.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.612946  4414 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:26.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.613379  4409 heartbeater.cc:461] Registering TS with master...
08:20:26.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.614403  4409 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:26.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.617202  3204 ts_manager.cc:194] Registered new tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:20:26.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.617359  3478 ts_manager.cc:194] Registered new tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:20:26.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.620159  3478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.131:37275
08:20:26.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:26.630707  3122 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
08:20:26.643 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:20:26.645 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

08:20:26.656 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.3.12.190"
      port: 33597
    }
    bound_http_address {
      host: "127.3.12.190"
      port: 45813
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.3.12.189"
      port: 34545
    }
    bound_http_address {
      host: "127.3.12.189"
      port: 39169
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.3.12.188"
      port: 42903
    }
    bound_http_address {
      host: "127.3.12.188"
      port: 37911
    }
  }
}

08:20:26.661 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

08:20:26.664 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.3.12.129"
      port: 46667
    }
    bound_http_address {
      host: "127.3.12.129"
      port: 38639
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.3.12.130"
      port: 37191
    }
    bound_http_address {
      host: "127.3.12.130"
      port: 39405
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.3.12.131"
      port: 42197
    }
    bound_http_address {
      host: "127.3.12.131"
      port: 42909
    }
  }
}

08:20:26.665 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
08:20:26.884 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
08:20:26.948 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
08:20:27.154 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.12.188:42903(127.3.12.188:42903)] connecting to peer
08:20:27.226 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.12.189:34545(127.3.12.189:34545)] connecting to peer
08:20:27.236 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.12.190:33597(127.3.12.190:33597)] connecting to peer
08:20:27.259 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.3.12.189:34545(127.3.12.189:34545)] Successfully connected to peer
08:20:27.262 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.3.12.188:42903(127.3.12.188:42903)] Successfully connected to peer
08:20:27.264 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.3.12.190:33597(127.3.12.190:33597)] Successfully connected to peer
08:20:27.380 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.3.12.190
08:20:27.380 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.3.12.189
08:20:27.380 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.3.12.188
08:20:27.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.625567  4409 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:27.670 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xaf625bc3, L:/127.0.0.1:44908 - R:/127.3.12.188:42903] using SASL/PLAIN
08:20:27.670 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x8b23611e, L:/127.0.0.1:51986 - R:/127.3.12.190:33597] using SASL/PLAIN
08:20:27.670 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xc54b1a38, L:/127.0.0.1:53802 - R:/127.3.12.189:34545] using SASL/PLAIN
08:20:27.713 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
08:20:27.714 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=604), Traces: [0ms] refreshing cache from master, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.12.188:42903, [111ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.12.189:34545, [121ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.12.190:33597, [573ms] Sub RPC ConnectToMaster: received response from server master-127.3.12.189:34545: OK, [573ms] Sub RPC ConnectToMaster: received response from server master-127.3.12.190:33597: OK, [573ms] Sub RPC ConnectToMaster: received response from server master-127.3.12.188:42903: OK, deferred=null) after lookup
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.774499  3478 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44908:
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097"
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:20:27.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:20:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:20:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:20:27.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:27.781472  3478 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
08:20:27.879 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table ca8cc32f66964bb4b72ceb4f76755443
08:20:27.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.922169  4295 tablet_service.cc:1511] Processing CreateTablet for tablet 203fe5a2ea01428e964d956caebbbf3c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097 [id=ca8cc32f66964bb4b72ceb4f76755443]), partition=RANGE (key) PARTITION UNBOUNDED
08:20:27.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.925233  4295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203fe5a2ea01428e964d956caebbbf3c. 1 dirs total, 0 dirs full, 0 dirs failed
08:20:27.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.926136  3773 tablet_service.cc:1511] Processing CreateTablet for tablet 203fe5a2ea01428e964d956caebbbf3c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097 [id=ca8cc32f66964bb4b72ceb4f76755443]), partition=RANGE (key) PARTITION UNBOUNDED
08:20:27.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.928130  3773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203fe5a2ea01428e964d956caebbbf3c. 1 dirs total, 0 dirs full, 0 dirs failed
08:20:27.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.932003  4032 tablet_service.cc:1511] Processing CreateTablet for tablet 203fe5a2ea01428e964d956caebbbf3c (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097 [id=ca8cc32f66964bb4b72ceb4f76755443]), partition=RANGE (key) PARTITION UNBOUNDED
08:20:27.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.934114  4032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203fe5a2ea01428e964d956caebbbf3c. 1 dirs total, 0 dirs full, 0 dirs failed
08:20:27.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.962726  4478 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap starting.
08:20:27.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.965194  4479 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap starting.
08:20:27.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.967505  4480 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap starting.
08:20:27.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.973075  4479 tablet_bootstrap.cc:654] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Neither blocks nor log segments found. Creating new log.
08:20:27.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.974328  4478 tablet_bootstrap.cc:654] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Neither blocks nor log segments found. Creating new log.
08:20:27.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.975961  4479 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Log is configured to *not* fsync() on all Append() calls
08:20:27.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.976854  4480 tablet_bootstrap.cc:654] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Neither blocks nor log segments found. Creating new log.
08:20:27.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.979297  4478 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Log is configured to *not* fsync() on all Append() calls
08:20:27.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.979379  4480 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Log is configured to *not* fsync() on all Append() calls
08:20:27.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.984516  4479 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: No bootstrap required, opened a new log
08:20:27.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.985199  4479 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.007s
08:20:27.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.986411  4480 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: No bootstrap required, opened a new log
08:20:27.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.987046  4480 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.014s
08:20:27.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.990067  4478 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: No bootstrap required, opened a new log
08:20:27.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:27.990724  4478 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent bootstrapping tablet: real 0.029s	user 0.008s	sys 0.011s
08:20:28.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.014240  4479 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.015264  4479 raft_consensus.cc:385] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:20:28.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.015620  4479 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09d29cb0944901a7b67274723e6a6b, State: Initialized, Role: FOLLOWER
08:20:28.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.016860  4479 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.016989  4480 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.017870  4480 raft_consensus.cc:385] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:20:28.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.018309  4480 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Initialized, Role: FOLLOWER
08:20:28.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.019902  4480 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.022065  4479 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent starting tablet: real 0.037s	user 0.036s	sys 0.002s
08:20:28.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.028144  4478 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.029198  4478 raft_consensus.cc:385] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:20:28.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.029547  4478 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Initialized, Role: FOLLOWER
08:20:28.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.030602  4478 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.042512  4480 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent starting tablet: real 0.055s	user 0.039s	sys 0.000s
08:20:28.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.050972  4478 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent starting tablet: real 0.060s	user 0.030s	sys 0.021s
08:20:28.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.065940  4488 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:20:28.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.066711  4488 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.070230  4488 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:28.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.084546  4327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b" is_pre_election: true
08:20:28.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.085701  4327 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 0.
08:20:28.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.087250  3809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da" is_pre_election: true
08:20:28.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.087409  3944 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127; no voters: 
08:20:28.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.088100  3809 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 0.
08:20:28.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.088342  4488 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Leader pre-election won for term 1
08:20:28.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.088681  4488 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:20:28.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.089023  4488 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 0 FOLLOWER]: Advancing to term 1
08:20:28.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.095443  4488 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:28.099236  4424 tablet.cc:2406] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:20:28.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.099423  4327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:20:28.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.099712  3809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:28.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.099984  4327 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 0 FOLLOWER]: Advancing to term 1
08:20:28.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.100247  3809 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 0 FOLLOWER]: Advancing to term 1
08:20:28.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.098555  4488 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 1 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:28.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.105621  3809 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 1.
08:20:28.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.106756  3944 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da; no voters: 
08:20:28.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.107545  4488 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 1 FOLLOWER]: Leader election won for term 1
08:20:28.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.108049  4327 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 1.
08:20:28.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.110359  4488 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 1 LEADER]: Becoming Leader. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Running, Role: LEADER
08:20:28.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.111479  4488 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:28.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.125911  3477 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130). New cstate: current_term: 1 leader_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: HEALTHY } } }
08:20:28.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:28.146178  4153 tablet.cc:2406] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:20:28.243 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table ca8cc32f66964bb4b72ceb4f76755443
08:20:28.270 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:28.276 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:28.278 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [13ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=null) after lookup
08:20:28.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:28.289237  3907 tablet.cc:2406] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:20:28.289 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:28.292 [DEBUG - kudu-nio-3] (Connection.java:757) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] Successfully connected to peer
08:20:28.376 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x2155adfb, L:/127.0.0.1:55072 - R:/127.3.12.130:37191] using TOKEN/null
08:20:28.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.396541  4327 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 1 FOLLOWER]: Refusing update from remote peer ea5f3bbdb1704cba8076e499c9ed1127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:20:28.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.396605  3809 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 1 FOLLOWER]: Refusing update from remote peer ea5f3bbdb1704cba8076e499c9ed1127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:20:28.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.398587  4488 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:20:28.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.399708  4498 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:20:28.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.435869  4504 mvcc.cc:204] Tried to move back new op lower bound from 7317597505091112960 to 7317597503953907712. Current Snapshot: MvccSnapshot[applied={T|T < 7317597505091112960}]
08:20:28.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.500278  4510 mvcc.cc:204] Tried to move back new op lower bound from 7317597505091112960 to 7317597503953907712. Current Snapshot: MvccSnapshot[applied={T|T < 7317597505091112960}]
08:20:28.685 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:20:28.686 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.130:37191
08:20:28.688 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:20:28.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:28.689695  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 3901
08:20:28.734 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:28.735 [DEBUG - kudu-nio-3] (Connection.java:239) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] handling channelInactive
08:20:28.735 [DEBUG - kudu-nio-3] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state READY due to: connection closed
08:20:28.752 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:28.755 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.755 [DEBUG - kudu-nio-4] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.756 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.777 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:28.777 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:28.778 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=3, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [21ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [21ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@867801192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2e849ba6 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59816398 -> wakeup thread Time-limited test)) after lookup
08:20:28.779 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:28.782 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.783 [DEBUG - kudu-nio-5] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.783 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.817 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:28.817 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:28.818 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=5, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=82), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [21ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [21ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [44ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [48ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [48ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@867801192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2e849ba6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59816398 -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974))) after lookup
08:20:28.820 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:28.823 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.823 [DEBUG - kudu-nio-6] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.824 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.857 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:28.857 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:28.858 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=7, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=122), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [21ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [21ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [44ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [48ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [48ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [84ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [89ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [89ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [122ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@867801192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2e849ba6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59816398 -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232))) after lookup
08:20:28.859 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:28.862 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.862 [DEBUG - kudu-nio-7] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.863 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.996 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:28.997 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:28.997 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=9, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=262), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [21ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [21ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [44ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [48ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [48ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [84ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [89ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [89ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [122ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [124ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [128ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [128ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [256ms] refreshing cache from master, [256ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [261ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@867801192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2e849ba6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59816398 -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423))) after lookup
08:20:28.998 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:28.999 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:28.999 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:29.000 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:29.538 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:29.538 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:29.539 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=11, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=803), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [21ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [21ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [44ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [48ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [48ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [84ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [89ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [89ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [122ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [124ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [128ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [128ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [256ms] refreshing cache from master, [256ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [261ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [263ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [264ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [265ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [796ms] refreshing cache from master, [796ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [803ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@867801192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2e849ba6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423) -> (continuation of Deferred@2058174149 after retry RPC@561957484), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59816398 -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423) -> (continuation of Deferred@2058174149 after retry RPC@561957484))) after lookup
08:20:29.541 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:29.542 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:29.542 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:29.543 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:29.857 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:29.858 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:29.859 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=13, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1123), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [21ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [21ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [44ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [48ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [48ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [84ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [89ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [89ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [122ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [124ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [128ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [128ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [256ms] refreshing cache from master, [256ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [261ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [263ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [264ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [265ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [796ms] refreshing cache from master, [796ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [803ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [805ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [807ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [808ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [1116ms] refreshing cache from master, [1116ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1122ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@867801192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2e849ba6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423) -> (continuation of Deferred@2058174149 after retry RPC@561957484) -> (continuation of Deferred@602324613 after retry RPC@1948007743), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59816398 -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423) -> (continuation of Deferred@2058174149 after retry RPC@561957484) -> (continuation of Deferred@602324613 after retry RPC@1948007743))) after lookup
08:20:29.860 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:29.862 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:29.862 [DEBUG - kudu-nio-2] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:29.862 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:30.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.079802  4532 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:20:30.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.080636  4532 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:30.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.083829  4532 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:30.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.089558  3683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:20:30.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.090003  4540 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:20:30.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.090759  4540 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:30.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.095891  4540 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:30.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.101442  3683 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:20:30.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.108040  4326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b" is_pre_election: true
08:20:30.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.108805  4209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:20:30.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.108873  4326 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 1.
08:20:30.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.112679  3682 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:20:30.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.114223  4532 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 1 FOLLOWER]: Leader pre-election won for term 2
08:20:30.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.114638  4532 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 1 FOLLOWER]: Starting leader election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:20:30.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.115034  4532 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 1 FOLLOWER]: Advancing to term 2
08:20:30.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.115171  4209 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:20:30.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.119522  3809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "8d09d29cb0944901a7b67274723e6a6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da" is_pre_election: true
08:20:30.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.122758  4532 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:30.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.124094  3809 raft_consensus.cc:2393] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d09d29cb0944901a7b67274723e6a6b in current term 2: Already voted for candidate f80bc8516214471ca7ea7896026da4da in this term.
08:20:30.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.125864  4532 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 2 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:30.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.126196  4326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:20:30.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.126782  4326 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 1 FOLLOWER]: Advancing to term 2
08:20:30.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.130749  4207 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b; no voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da
08:20:30.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.132485  3683 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:20:30.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.134074  4326 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 2.
08:20:30.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.134586  4540 raft_consensus.cc:2749] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
08:20:30.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.135702  3682 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:20:30.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.136758  4532 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Leader election won for term 2
08:20:30.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.138897  4532 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 LEADER]: Becoming Leader. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Running, Role: LEADER
08:20:30.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.140232  4532 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:30.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.151747  3478 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da reported cstate change: term changed from 1 to 2, leader changed from ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130) to f80bc8516214471ca7ea7896026da4da (127.3.12.129). New cstate: current_term: 2 leader_uuid: "f80bc8516214471ca7ea7896026da4da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: UNKNOWN } } }
08:20:30.238 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:30.238 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:30.239 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=15, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1503), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [21ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [21ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [44ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [48ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [48ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [84ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [89ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [89ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [122ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [124ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [128ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [128ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [256ms] refreshing cache from master, [256ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [261ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [263ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [264ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [265ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [796ms] refreshing cache from master, [796ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [803ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [805ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [807ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [808ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [1116ms] refreshing cache from master, [1116ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1122ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [1124ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1127ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [1127ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [1496ms] refreshing cache from master, [1496ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1502ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@867801192(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2e849ba6 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423) -> (continuation of Deferred@2058174149 after retry RPC@561957484) -> (continuation of Deferred@602324613 after retry RPC@1948007743) -> (continuation of Deferred@1220786964 after retry RPC@1562608374), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59816398 -> wakeup thread Time-limited test -> (continuation of Deferred@1514056113 after retry RPC@420796974) -> (continuation of Deferred@484875008 after retry RPC@443498232) -> (continuation of Deferred@470527686 after retry RPC@1176530423) -> (continuation of Deferred@2058174149 after retry RPC@561957484) -> (continuation of Deferred@602324613 after retry RPC@1948007743) -> (continuation of Deferred@1220786964 after retry RPC@1562608374))) after lookup
08:20:30.240 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:30.242 [DEBUG - kudu-nio-3] (Connection.java:757) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] Successfully connected to peer
08:20:30.277 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xa8833bf7, L:/127.0.0.1:40118 - R:/127.3.12.129:46667] using TOKEN/null
08:20:30.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.291291  4326 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Refusing update from remote peer f80bc8516214471ca7ea7896026da4da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
08:20:30.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.294279  4553 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
08:20:30.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.295827  3683 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da -> Peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Couldn't send request to peer ea5f3bbdb1704cba8076e499c9ed1127. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:20:30.372 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.130:37191
08:20:30.374 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:20:30.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:30.376120  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:30.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:30.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:30.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:30.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:30.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.867024  4594 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:30.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.868202  4594 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:30.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.869531  4594 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:30.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.930471  4594 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:30.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:30.932183  4594 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.130
08:20:31.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:31.000560  4594 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:31.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:20:31.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:31.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:31.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:31.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:31.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:31.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:31.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:20:31.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:31.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:31.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:31.004760  4594 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:31.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:31.008803  4594 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:31.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:31.025174  4608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:32.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:31.026032  4611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:32.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:32.738145  4594 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.713s	user 0.644s	sys 1.052s
08:20:32.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:32.428822  4607 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee40 after lost signal to thread 4594
08:20:32.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:32.740724  4594 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.716s	user 0.644s	sys 1.052s
08:20:32.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:32.741055  4613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:32.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:32.744867  4612 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1719 milliseconds
08:20:32.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.744940  4594 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:32.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.746690  4594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:32.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.749475  4594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:32.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.751338  4594 hybrid_clock.cc:648] HybridClock initialized: now 1786522832751287 us; error 58 us; skew 500 ppm
08:20:32.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.752296  4594 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:32.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.760202  4594 webserver.cc:533] Webserver started at http://127.3.12.130:39405/ using document root <none> and password file <none>
08:20:32.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.761258  4594 fs_manager.cc:362] Metadata directory not provided
08:20:32.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.761497  4594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:32.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.770130  4594 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.006s	sys 0.001s
08:20:32.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.775496  4639 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:32.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.777176  4594 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.005s
08:20:32.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.777550  4594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:32.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:20:32.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:23 on dist-test-slave-t7g5"
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.779696  4594 fs_report.cc:389] FS layout report
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/data
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:32.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:32.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.863451  4594 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:32.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.865130  4594 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:32.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.865594  4594 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:32.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.868979  4594 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:32.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.876489  4651 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:20:32.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.886472  4594 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:20:32.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.886775  4594 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.000s	sys 0.003s
08:20:32.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.887099  4594 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:20:32.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.893270  4594 ts_tablet_manager.cc:616] Registered 1 tablets
08:20:32.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.893496  4594 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.003s	sys 0.000s
08:20:32.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.893955  4651 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap starting.
08:20:32.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.960976  4651 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Log is configured to *not* fsync() on all Append() calls
08:20:32.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:32.965315  3683 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da -> Peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Couldn't send request to peer ea5f3bbdb1704cba8076e499c9ed1127. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:20:32.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.998158  4651 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:20:33.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:32.999850  4651 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap complete.
08:20:33.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.001936  4651 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent bootstrapping tablet: real 0.108s	user 0.059s	sys 0.043s
08:20:33.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.025255  4651 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:33.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.028895  4651 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Initialized, Role: FOLLOWER
08:20:33.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.030507  4651 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:33.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.035316  4651 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent starting tablet: real 0.033s	user 0.027s	sys 0.003s
08:20:33.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.087702  4594 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.130:37191
08:20:33.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.087913  4842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.130:37191 every 8 connection(s)
08:20:33.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.090477  4594 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:33.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.097482  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 4594
08:20:33.098 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:33.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:33.129468  3391 debug-util.cc:398] Leaking SignalData structure 0x7b0800098080 after lost signal to thread 3278
08:20:33.137 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:20:33.138 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.129:46667
08:20:33.138 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:20:33.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.139698  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 3633
08:20:33.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.167173  4845 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:33.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.167747  4845 heartbeater.cc:461] Registering TS with master...
08:20:33.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.168823  4843 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:33.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.169193  4843 heartbeater.cc:461] Registering TS with master...
08:20:33.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.169252  4845 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:33.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.170361  4843 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:33.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.170461  4846 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:33.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.170850  4846 heartbeater.cc:461] Registering TS with master...
08:20:33.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.171780  4846 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:33.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.174207  3477 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:33.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.176625  3339 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:33.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.177359  3477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.130:56281
08:20:33.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:33.183013  3204 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:33.199 [DEBUG - kudu-nio-3] (Connection.java:239) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] handling channelInactive
08:20:33.199 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state READY due to: connection closed
08:20:33.200 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:33.202 [DEBUG - Time-limited test] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:33.203 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.203 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.203 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.217 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:33.218 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:33.218 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1411517609(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@155710e9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@26e40713 -> wakeup thread Time-limited test)) after lookup
08:20:33.219 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:33.220 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.221 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.221 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.238 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:33.239 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:33.240 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [20ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [20ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1411517609(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@155710e9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@26e40713 -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430))) after lookup
08:20:33.241 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:33.242 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.242 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.242 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.278 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:33.278 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:33.279 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [20ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [20ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1411517609(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@155710e9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@26e40713 -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794))) after lookup
08:20:33.280 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:33.282 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.282 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.283 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.317 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:33.317 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:33.318 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=116), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [20ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [20ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [79ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [81ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [82ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [115ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1411517609(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@155710e9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@26e40713 -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934))) after lookup
08:20:33.319 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:33.320 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.320 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.321 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.537 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:33.538 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:33.539 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=337), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [20ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [20ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [79ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [81ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [82ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [115ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [117ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [120ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [120ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [330ms] refreshing cache from master, [330ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [336ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1411517609(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@155710e9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934) -> (continuation of Deferred@1230245936 after retry RPC@398673855), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@26e40713 -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934) -> (continuation of Deferred@1230245936 after retry RPC@398673855))) after lookup
08:20:33.542 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:33.543 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.543 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.544 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.638 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:33.639 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:33.639 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=438), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [20ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [20ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [79ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [81ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [82ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [115ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [117ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [120ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [120ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [330ms] refreshing cache from master, [330ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [336ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [340ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [343ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [343ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [437ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1411517609(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@155710e9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934) -> (continuation of Deferred@1230245936 after retry RPC@398673855) -> (continuation of Deferred@1409476245 after retry RPC@1500626197), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@26e40713 -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934) -> (continuation of Deferred@1230245936 after retry RPC@398673855) -> (continuation of Deferred@1409476245 after retry RPC@1500626197))) after lookup
08:20:33.640 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:33.642 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.643 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:33.643 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:34.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.181948  4843 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:34.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.392290  4865 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f80bc8516214471ca7ea7896026da4da)
08:20:34.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.392765  4865 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:34.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.394595  4865 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:34.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:34.402938  4207 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:20:34.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.410386  4754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "8d09d29cb0944901a7b67274723e6a6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" is_pre_election: true
08:20:34.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.411209  4754 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d09d29cb0944901a7b67274723e6a6b in term 1.
08:20:34.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.412688  4209 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127; no voters: f80bc8516214471ca7ea7896026da4da
08:20:34.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.413437  4865 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Leader pre-election won for term 3
08:20:34.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.413810  4865 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Starting leader election (detected failure of leader f80bc8516214471ca7ea7896026da4da)
08:20:34.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.414146  4865 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 2 FOLLOWER]: Advancing to term 3
08:20:34.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.421258  4865 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:34.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.423501  4865 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 3 election: Requested vote from peers f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:34.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.425257  4754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "8d09d29cb0944901a7b67274723e6a6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:20:34.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.425881  4754 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 1 FOLLOWER]: Advancing to term 3
08:20:34.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:34.427656  4207 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:20:34.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.433079  4754 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d09d29cb0944901a7b67274723e6a6b in term 3.
08:20:34.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.434615  4209 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127; no voters: f80bc8516214471ca7ea7896026da4da
08:20:34.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.435487  4865 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 3 FOLLOWER]: Leader election won for term 3
08:20:34.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.437175  4865 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 3 LEADER]: Becoming Leader. State: Replica: 8d09d29cb0944901a7b67274723e6a6b, State: Running, Role: LEADER
08:20:34.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.439116  4865 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:34.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.449581  3477 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b reported cstate change: term changed from 2 to 3, leader changed from f80bc8516214471ca7ea7896026da4da (127.3.12.129) to 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131). New cstate: current_term: 3 leader_uuid: "8d09d29cb0944901a7b67274723e6a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: UNKNOWN } } }
08:20:34.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:34.902877  4207 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b -> Peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Couldn't send request to peer f80bc8516214471ca7ea7896026da4da. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:20:34.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.932186  4754 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 3 FOLLOWER]: Refusing update from remote peer 8d09d29cb0944901a7b67274723e6a6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
08:20:34.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:34.934607  4865 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
08:20:35.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:35.369808  4207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111) [suppressed 3 similar messages]
08:20:37.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.193608  4207 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b -> Peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Couldn't send request to peer f80bc8516214471ca7ea7896026da4da. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:20:37.358 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:37.359 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:37.359 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=4158), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [20ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [20ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [79ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [81ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [82ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [110ms] refreshing cache from master, [110ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [115ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [117ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [120ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [120ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [330ms] refreshing cache from master, [330ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [336ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [340ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [343ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [343ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [437ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [439ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [442ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [442ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [4150ms] refreshing cache from master, [4151ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [4156ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1411517609(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@155710e9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934) -> (continuation of Deferred@1230245936 after retry RPC@398673855) -> (continuation of Deferred@1409476245 after retry RPC@1500626197) -> (continuation of Deferred@64757192 after retry RPC@1469157811), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@26e40713 -> wakeup thread Time-limited test -> (continuation of Deferred@787700940 after retry RPC@673782430) -> (continuation of Deferred@1074763831 after retry RPC@546473794) -> (continuation of Deferred@592780631 after retry RPC@796263934) -> (continuation of Deferred@1230245936 after retry RPC@398673855) -> (continuation of Deferred@1409476245 after retry RPC@1500626197) -> (continuation of Deferred@64757192 after retry RPC@1469157811))) after lookup
08:20:37.360 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:37.361 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] Successfully connected to peer
08:20:37.385 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x0513afc9, L:/127.0.0.1:55780 - R:/127.3.12.131:42197] using TOKEN/null
08:20:37.446 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.129:46667
08:20:37.447 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:20:37.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:37.448378  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:37.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:37.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:37.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:20:37.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:46667
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38639
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:37.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.833562  4934 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:37.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.834211  4934 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.834832  4934 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:37.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.873517  4934 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:37.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.874557  4934 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.129
08:20:37.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:37.912986  4934 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:37.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:20:37.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:37.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:46667
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38639
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:37.915038  4934 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:37.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:37.916921  4934 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:37.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.931478  4943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:39.334479  4941 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee40 after lost signal to thread 4934
08:20:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:39.487394  4941 kernel_stack_watchdog.cc:198] Thread 4934 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
08:20:39.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:20:39.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:20:39.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:20:39.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:20:39.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:37.932726  4945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:39.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:39.492057  4934 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.560s	user 0.597s	sys 0.890s
08:20:39.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:39.492507  4934 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.561s	user 0.597s	sys 0.890s
08:20:39.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:39.493533  4946 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1559 milliseconds
08:20:39.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.494587  4934 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:39.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:39.494626  4949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:39.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.499845  4934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:39.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.503050  4934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.506692  4934 hybrid_clock.cc:648] HybridClock initialized: now 1786522839506638 us; error 72 us; skew 500 ppm
08:20:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.507869  4934 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:39.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.516024  4934 webserver.cc:533] Webserver started at http://127.3.12.129:38639/ using document root <none> and password file <none>
08:20:39.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.517391  4934 fs_manager.cc:362] Metadata directory not provided
08:20:39.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.517753  4934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:39.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.529152  4934 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.000s
08:20:39.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.536697  4956 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:39.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.538522  4934 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.005s	sys 0.001s
08:20:39.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.539031  4934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:20 on dist-test-slave-t7g5"
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.541733  4934 fs_report.cc:389] FS layout report
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/data
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:39.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:39.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:39.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:39.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:39.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:39.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.609344  4934 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.611253  4934 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:39.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.611821  4934 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:39.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.614835  4934 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:39.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.621855  4968 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:20:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.632480  4934 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:20:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.632815  4934 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.001s	sys 0.001s
08:20:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.633144  4934 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:20:39.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.639889  4934 ts_tablet_manager.cc:616] Registered 1 tablets
08:20:39.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.640117  4934 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.003s	sys 0.003s
08:20:39.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.640892  4968 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap starting.
08:20:39.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:39.657871  4207 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b -> Peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Couldn't send request to peer f80bc8516214471ca7ea7896026da4da. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
08:20:39.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.704838  4968 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Log is configured to *not* fsync() on all Append() calls
08:20:39.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.758527  4968 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:20:39.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.759701  4968 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap complete.
08:20:39.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.761626  4968 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent bootstrapping tablet: real 0.121s	user 0.078s	sys 0.035s
08:20:39.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.786697  4968 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:39.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.790481  4968 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Initialized, Role: FOLLOWER
08:20:39.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.792557  4968 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:39.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.797443  4968 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent starting tablet: real 0.035s	user 0.029s	sys 0.004s
08:20:39.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.834306  4934 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.129:46667
08:20:39.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.834674  5160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.129:46667 every 8 connection(s)
08:20:39.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.837298  4934 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:39.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.845304  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 4934
08:20:39.846 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:39.873 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:20:39.875 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.131:42197
08:20:39.876 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:20:39.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.876564  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 4170
08:20:39.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.886340  5161 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:39.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.886906  5161 heartbeater.cc:461] Registering TS with master...
08:20:39.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.888229  5161 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:39.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.892521  3477 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:39.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.896807  3477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.129:45419
08:20:39.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.910717  5162 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:39.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.911142  5162 heartbeater.cc:461] Registering TS with master...
08:20:39.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.911855  5162 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:39.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.912336  5163 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:39.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.912667  5163 heartbeater.cc:461] Registering TS with master...
08:20:39.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.913280  5163 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:39.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.916659  3339 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:39.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:39.916853  3204 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:39.942 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] handling channelInactive
08:20:39.942 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state READY due to: connection closed
08:20:39.943 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:39.944 [DEBUG - Time-limited test] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:39.946 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.946 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.946 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.956 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:39.957 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:39.957 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1840692707(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@36f1a5ef -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5dc91f1a -> wakeup thread Time-limited test)) after lookup
08:20:39.958 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:39.959 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.959 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.959 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.976 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:39.977 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:39.977 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [15ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [15ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1840692707(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@36f1a5ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5dc91f1a -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663))) after lookup
08:20:39.978 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:39.979 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.979 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.979 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:39.997 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:39.997 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:39.998 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=54), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [15ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [15ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [34ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [35ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [35ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1840692707(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@36f1a5ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5dc91f1a -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918))) after lookup
08:20:39.999 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:40.000 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.000 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.001 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.076 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:40.077 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:40.077 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=133), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [15ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [15ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [34ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [35ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [35ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [54ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [57ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [57ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [127ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [132ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1840692707(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@36f1a5ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5dc91f1a -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962))) after lookup
08:20:40.078 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:40.079 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.080 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.080 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.157 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:40.157 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:40.158 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=214), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [15ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [15ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [34ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [35ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [35ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [54ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [57ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [57ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [127ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [132ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [134ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [136ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [136ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [212ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1840692707(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@36f1a5ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962) -> (continuation of Deferred@1677625078 after retry RPC@85595955), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5dc91f1a -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962) -> (continuation of Deferred@1677625078 after retry RPC@85595955))) after lookup
08:20:40.158 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:40.159 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.160 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.160 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:40.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:40.902837  5161 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:41.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:41.018 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:41.019 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1075), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [15ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [15ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [34ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [35ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [35ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [54ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [57ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [57ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [127ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [132ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [134ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [136ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [136ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [212ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [214ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [216ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [216ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [1067ms] refreshing cache from master, [1068ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1073ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1840692707(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@36f1a5ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962) -> (continuation of Deferred@1677625078 after retry RPC@85595955) -> (continuation of Deferred@1299291545 after retry RPC@411547787), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5dc91f1a -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962) -> (continuation of Deferred@1677625078 after retry RPC@85595955) -> (continuation of Deferred@1299291545 after retry RPC@411547787))) after lookup
08:20:41.020 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:41.022 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:41.022 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:41.022 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:41.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.247871  5174 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:20:41.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.248502  5174 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:41.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.251809  5174 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:41.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:41.256140  4958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:41.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:41.260546  4958 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:41.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.265349  4754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" is_pre_election: true
08:20:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.267481  4959 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127
08:20:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.268299  5174 raft_consensus.cc:2749] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
08:20:41.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.320039  5185 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8d09d29cb0944901a7b67274723e6a6b)
08:20:41.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.320573  5185 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:41.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.323509  5185 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:41.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:41.327080  4640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:41.331908  4640 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:41.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.337617  5091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da" is_pre_election: true
08:20:41.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.338450  5091 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 2.
08:20:41.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.339964  4640 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b
08:20:41.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.340767  5185 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 3 FOLLOWER]: Leader pre-election won for term 4
08:20:41.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.341120  5185 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8d09d29cb0944901a7b67274723e6a6b)
08:20:41.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.341375  5185 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 3 FOLLOWER]: Advancing to term 4
08:20:41.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.346328  5185 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:41.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.347981  5185 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 4 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:41.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.350032  5091 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:41.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.350673  5091 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 2 FOLLOWER]: Advancing to term 4
08:20:41.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:41.351694  4640 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:41.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.359663  5091 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 4.
08:20:41.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.361159  4640 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b
08:20:41.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.362152  5185 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 4 FOLLOWER]: Leader election won for term 4
08:20:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.363813  5185 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 4 LEADER]: Becoming Leader. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Running, Role: LEADER
08:20:41.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.364991  5185 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:41.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.378422  3477 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 reported cstate change: term changed from 3 to 4, leader changed from 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131) to ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130). New cstate: current_term: 4 leader_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: HEALTHY } } }
08:20:41.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.923722  5091 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 4 FOLLOWER]: Refusing update from remote peer ea5f3bbdb1704cba8076e499c9ed1127: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
08:20:41.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.925575  5215 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
08:20:41.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:41.947655  5220 mvcc.cc:204] Tried to move back new op lower bound from 7317597541961371648 to 7317597529871290368. Current Snapshot: MvccSnapshot[applied={T|T < 7317597541961371648}]
08:20:41.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:41.985756  4640 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 -> Peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Couldn't send request to peer 8d09d29cb0944901a7b67274723e6a6b. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:20:41.996 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:41.997 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:41.997 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2053), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [15ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [15ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [34ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [35ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [35ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [54ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [57ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [57ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [127ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [132ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [134ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [136ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [136ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [212ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [214ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [216ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [216ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [1067ms] refreshing cache from master, [1068ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1073ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [1075ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [1078ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [1078ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2047ms] refreshing cache from master, [2047ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [2052ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1840692707(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@36f1a5ef -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962) -> (continuation of Deferred@1677625078 after retry RPC@85595955) -> (continuation of Deferred@1299291545 after retry RPC@411547787) -> (continuation of Deferred@509899512 after retry RPC@990040671), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5dc91f1a -> wakeup thread Time-limited test -> (continuation of Deferred@250059118 after retry RPC@465794663) -> (continuation of Deferred@2114255751 after retry RPC@2129975918) -> (continuation of Deferred@571600932 after retry RPC@2062444962) -> (continuation of Deferred@1677625078 after retry RPC@85595955) -> (continuation of Deferred@1299291545 after retry RPC@411547787) -> (continuation of Deferred@509899512 after retry RPC@990040671))) after lookup
08:20:41.999 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:42.001 [DEBUG - kudu-nio-3] (Connection.java:757) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] Successfully connected to peer
08:20:42.032 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xe8c35dfe, L:/127.0.0.1:33408 - R:/127.3.12.130:37191] using TOKEN/null
08:20:42.089 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.131:42197
08:20:42.090 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:20:42.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:42.092336  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:42197
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:20:42.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:42.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42909
08:20:42.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:20:42.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:42.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:42.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:42.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:42.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.480931  5244 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:42.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.481572  5244 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:42.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.482317  5244 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:42.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.521808  5244 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:42.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.523064  5244 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.131
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:42.563930  5244 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:42.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:42197
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42909
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:42.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:42.565450  5244 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:42.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:42.567504  5244 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:42.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.583139  5252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.584456  5255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:42.588526  5260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:43.747613  5258 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
08:20:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.747772  5244 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:43.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.751982  5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:43.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.754540  5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:43.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.756038  5244 hybrid_clock.cc:648] HybridClock initialized: now 1786522843755970 us; error 104 us; skew 500 ppm
08:20:43.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.757062  5244 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:43.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.765411  5244 webserver.cc:533] Webserver started at http://127.3.12.131:42909/ using document root <none> and password file <none>
08:20:43.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.766940  5244 fs_manager.cc:362] Metadata directory not provided
08:20:43.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.767246  5244 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:43.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.776281  5244 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.004s
08:20:43.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.781870  5278 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:43.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.783232  5244 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
08:20:43.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.783578  5244 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:43.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:20:43.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:26 on dist-test-slave-t7g5"
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.785780  5244 fs_report.cc:389] FS layout report
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/data
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:43.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:43.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.842217  5244 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:43.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.843967  5244 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:43.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.844501  5244 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:43.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.847823  5244 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:43.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.854817  5295 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:20:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.863760  5244 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:20:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.864108  5244 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.003s
08:20:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.864398  5244 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:20:43.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.869874  5244 ts_tablet_manager.cc:616] Registered 1 tablets
08:20:43.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.870167  5244 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.001s
08:20:43.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.870637  5295 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap starting.
08:20:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.939575  5295 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Log is configured to *not* fsync() on all Append() calls
08:20:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.993929  5295 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:20:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.995712  5295 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap complete.
08:20:44.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:43.998533  5295 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent bootstrapping tablet: real 0.128s	user 0.075s	sys 0.049s
08:20:44.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.029840  5295 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:44.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.033699  5295 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09d29cb0944901a7b67274723e6a6b, State: Initialized, Role: FOLLOWER
08:20:44.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.035565  5295 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:44.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.041244  5295 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent starting tablet: real 0.042s	user 0.033s	sys 0.007s
08:20:44.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.080371  5244 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.131:42197
08:20:44.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.080677  5495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.131:42197 every 8 connection(s)
08:20:44.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.083185  5244 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:44.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.084081  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 5244
08:20:44.084 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:44.123 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:20:44.139 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.130:37191
08:20:44.140 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:20:44.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.140882  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 4594
08:20:44.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.156690  5500 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:44.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.157495  5500 heartbeater.cc:461] Registering TS with master...
08:20:44.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.159529  5500 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:44.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.167083  3339 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:20:44.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.205546  5503 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:44.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.206004  5503 heartbeater.cc:461] Registering TS with master...
08:20:44.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.208133  5503 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:44.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.212303  3204 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:20:44.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.218478  5497 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:44.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.218897  5497 heartbeater.cc:461] Registering TS with master...
08:20:44.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.219884  5497 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:44.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.223506  3478 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:20:44.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:44.225925  3478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.131:52393
08:20:44.229 [DEBUG - kudu-nio-3] (Connection.java:239) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] handling channelInactive
08:20:44.231 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:44.231 [DEBUG - kudu-nio-3] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state READY due to: connection closed
08:20:44.233 [DEBUG - Time-limited test] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:44.234 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.234 [DEBUG - kudu-nio-4] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.234 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.256 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:44.257 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:44.257 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@345143899(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@392fd2bc -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bd0cfe0 -> wakeup thread Time-limited test)) after lookup
08:20:44.258 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:44.259 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.259 [DEBUG - kudu-nio-5] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.259 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.276 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:44.276 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:44.276 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [25ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [27ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [27ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@345143899(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@392fd2bc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bd0cfe0 -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095))) after lookup
08:20:44.277 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:44.279 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.279 [DEBUG - kudu-nio-6] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.279 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.316 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:44.316 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:44.316 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [25ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [27ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [27ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [45ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [47ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [47ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@345143899(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@392fd2bc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bd0cfe0 -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923))) after lookup
08:20:44.317 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:44.319 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.319 [DEBUG - kudu-nio-7] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.319 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.356 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:44.357 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:44.357 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=125), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [25ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [27ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [27ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [45ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [47ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [47ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [85ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [87ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [87ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [124ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@345143899(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@392fd2bc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bd0cfe0 -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905))) after lookup
08:20:44.359 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:44.360 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.360 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.360 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.678 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:44.678 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:44.678 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=446), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [25ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [27ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [27ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [45ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [47ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [47ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [85ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [87ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [87ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [124ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [126ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [128ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [128ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [439ms] refreshing cache from master, [440ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [445ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@345143899(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@392fd2bc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905) -> (continuation of Deferred@1472277236 after retry RPC@1170141910), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bd0cfe0 -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905) -> (continuation of Deferred@1472277236 after retry RPC@1170141910))) after lookup
08:20:44.679 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:44.680 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.680 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.680 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.717 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:44.717 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:44.718 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=486), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [25ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [27ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [27ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [45ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [47ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [47ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [85ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [87ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [87ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [124ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [126ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [128ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [128ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [439ms] refreshing cache from master, [440ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [445ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [447ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [448ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [448ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [479ms] refreshing cache from master, [479ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [484ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@345143899(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@392fd2bc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905) -> (continuation of Deferred@1472277236 after retry RPC@1170141910) -> (continuation of Deferred@990984099 after retry RPC@160123477), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bd0cfe0 -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905) -> (continuation of Deferred@1472277236 after retry RPC@1170141910) -> (continuation of Deferred@990984099 after retry RPC@160123477))) after lookup
08:20:44.719 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:44.720 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.720 [DEBUG - kudu-nio-2] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:44.720 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:20:45.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.232192  5497 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:45.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.610183  5519 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:20:45.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.610708  5519 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:45.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.612586  5519 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:45.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:45.619344  4959 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:20:45.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.625335  5410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b" is_pre_election: true
08:20:45.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.626158  5410 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 3.
08:20:45.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.628217  4958 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:20:45.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.629354  5519 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 4 FOLLOWER]: Leader pre-election won for term 5
08:20:45.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.629776  5519 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 4 FOLLOWER]: Starting leader election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:20:45.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.630049  5519 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 4 FOLLOWER]: Advancing to term 5
08:20:45.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.636250  5519 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:45.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.638309  5519 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 5 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:45.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.639561  5410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:20:45.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.640242  5410 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 3 FOLLOWER]: Advancing to term 5
08:20:45.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:45.643419  4959 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:20:45.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.648061  5410 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 5.
08:20:45.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.649374  4958 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:20:45.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.650257  5519 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Leader election won for term 5
08:20:45.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.651722  5519 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 LEADER]: Becoming Leader. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Running, Role: LEADER
08:20:45.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.652843  5519 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:45.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:45.662002  3478 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da reported cstate change: term changed from 4 to 5, leader changed from ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130) to f80bc8516214471ca7ea7896026da4da (127.3.12.129). New cstate: current_term: 5 leader_uuid: "f80bc8516214471ca7ea7896026da4da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: UNKNOWN } } }
08:20:46.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:46.134285  4959 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da -> Peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Couldn't send request to peer ea5f3bbdb1704cba8076e499c9ed1127. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:20:46.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:46.210356  5410 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 5 FOLLOWER]: Refusing update from remote peer f80bc8516214471ca7ea7896026da4da: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
08:20:46.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:46.212100  5519 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
08:20:46.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:46.593783  4959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111) [suppressed 3 similar messages]
08:20:46.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:46.797719  3391 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0a60 after lost signal to thread 3278
08:20:46.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:46.798301  3391 debug-util.cc:398] Leaking SignalData structure 0x7b0800050260 after lost signal to thread 3396
08:20:48.357 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:48.357 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:48.358 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=4125), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [25ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [27ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [27ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [45ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [47ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [47ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [85ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [87ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [87ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [124ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [126ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [128ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [128ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [439ms] refreshing cache from master, [440ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [445ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [447ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [448ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [448ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [479ms] refreshing cache from master, [479ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [484ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [486ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [488ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [488ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [4119ms] refreshing cache from master, [4119ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [4124ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@345143899(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@392fd2bc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905) -> (continuation of Deferred@1472277236 after retry RPC@1170141910) -> (continuation of Deferred@990984099 after retry RPC@160123477) -> (continuation of Deferred@923881849 after retry RPC@1687907372), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bd0cfe0 -> wakeup thread Time-limited test -> (continuation of Deferred@59729857 after retry RPC@237832095) -> (continuation of Deferred@1313850266 after retry RPC@674767923) -> (continuation of Deferred@840323516 after retry RPC@1541857905) -> (continuation of Deferred@1472277236 after retry RPC@1170141910) -> (continuation of Deferred@990984099 after retry RPC@160123477) -> (continuation of Deferred@923881849 after retry RPC@1687907372))) after lookup
08:20:48.364 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:48.366 [DEBUG - kudu-nio-3] (Connection.java:757) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] Successfully connected to peer
08:20:48.398 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xaeb41235, L:/127.0.0.1:57902 - R:/127.3.12.129:46667] using TOKEN/null
08:20:48.456 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.130:37191
08:20:48.457 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:48.459889  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:48.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.535427  4959 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da -> Peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Couldn't send request to peer ea5f3bbdb1704cba8076e499c9ed1127. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:20:48.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.821863  5614 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:48.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.822436  5614 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:48.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.823066  5614 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:48.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.858565  5614 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:48.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.859609  5614 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.130
08:20:48.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:48.902220  5614 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:48.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:20:48.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:48.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:48.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:48.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:48.904011  5614 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:48.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:48.906085  5614 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:48.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.921263  5624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:50.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:48.923094  5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:50.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:50.236027  5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:50.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:50.240571  5614 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.319s	user 0.518s	sys 0.755s
08:20:50.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:50.241044  5614 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.320s	user 0.519s	sys 0.756s
08:20:50.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:50.241365  5626 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1312 milliseconds
08:20:50.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.242193  5614 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.243508  5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.246701  5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:50.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.248272  5614 hybrid_clock.cc:648] HybridClock initialized: now 1786522850248241 us; error 74 us; skew 500 ppm
08:20:50.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.249486  5614 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:50.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.258911  5614 webserver.cc:533] Webserver started at http://127.3.12.130:39405/ using document root <none> and password file <none>
08:20:50.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.260136  5614 fs_manager.cc:362] Metadata directory not provided
08:20:50.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.260413  5614 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:50.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.270442  5614 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.005s	sys 0.001s
08:20:50.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.276929  5651 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:50.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.278267  5614 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.006s
08:20:50.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.278609  5614 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:23 on dist-test-slave-t7g5"
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.280839  5614 fs_report.cc:389] FS layout report
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/data
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:50.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:50.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:50.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:50.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:50.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.388834  5614 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:50.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.390625  5614 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:50.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.391167  5614 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:50.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.394074  5614 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:50.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.400985  5675 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:20:50.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.411444  5614 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:20:50.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.411746  5614 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.001s	sys 0.001s
08:20:50.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.412088  5614 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:20:50.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.418857  5614 ts_tablet_manager.cc:616] Registered 1 tablets
08:20:50.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.419142  5614 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.003s	sys 0.003s
08:20:50.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.419852  5675 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap starting.
08:20:50.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.483052  5675 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Log is configured to *not* fsync() on all Append() calls
08:20:50.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.565645  5675 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:20:50.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.567332  5675 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap complete.
08:20:50.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.569363  5675 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent bootstrapping tablet: real 0.150s	user 0.109s	sys 0.036s
08:20:50.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.594429  5675 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:50.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.597685  5675 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Initialized, Role: FOLLOWER
08:20:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.599431  5675 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.603605  5675 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent starting tablet: real 0.034s	user 0.031s	sys 0.000s
08:20:50.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.611292  5614 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.130:37191
08:20:50.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.611647  5889 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.130:37191 every 8 connection(s)
08:20:50.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.613893  5614 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:20:50.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.619767  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 5614
08:20:50.620 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:50.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.666181  5891 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:50.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.667008  5891 heartbeater.cc:461] Registering TS with master...
08:20:50.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.668561  5891 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:50.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.674948  3477 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:50.684 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:20:50.685 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.129:46667
08:20:50.686 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:20:50.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.686338  5896 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.686640  5895 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.686761  5896 heartbeater.cc:461] Registering TS with master...
08:20:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.686724  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 4934
08:20:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.686965  5895 heartbeater.cc:461] Registering TS with master...
08:20:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.687799  5895 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.687849  5896 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:50.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.692510  3339 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:50.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.699067  3477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.130:43251
08:20:50.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:50.705390  3204 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:50.737 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:50.738 [DEBUG - kudu-nio-3] (Connection.java:239) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] handling channelInactive
08:20:50.738 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state READY due to: connection closed
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:50.740186  5907 negotiation.cc:336] Failed RPC negotiation. Trace:
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.640008 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.3.12.129:50887 (local address 127.3.12.130:37191)
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.663418 (+ 23410us) server_negotiation.cc:207] Beginning negotiation
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.663451 (+    33us) server_negotiation.cc:400] Waiting for connection header
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.663514 (+    63us) server_negotiation.cc:408] Connection header received
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.663776 (+   262us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.663824 (+    48us) server_negotiation.cc:462] Received NEGOTIATE request from client
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.664338 (+   514us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.666310 (+  1972us) server_negotiation.cc:227] Negotiated authn=SASL
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.669423 (+  3113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.681104 (+ 11681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.684268 (+  3164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
08:20:50.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.685259 (+   991us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
08:20:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:20:50.737491 (+ 52232us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.129:50887: BlockingRecv error: recv got EOF from 127.3.12.129:50887 (error 108)
08:20:50.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":22575,"thread_start_us":22094,"threads_started":1}
08:20:50.751 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:50.753 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.753 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.753 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.776 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:50.776 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:50.776 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=3, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [15ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [15ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@112167106(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@39ea6e37 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@a687359 -> wakeup thread Time-limited test)) after lookup
08:20:50.777 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:50.778 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.779 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.779 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.796 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:50.796 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:50.797 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=5, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [15ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [15ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@112167106(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@39ea6e37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@a687359 -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643))) after lookup
08:20:50.798 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:50.799 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.800 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.800 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.856 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:50.856 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:50.857 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=7, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=118), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [15ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [15ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [59ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [62ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [62ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [118ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@112167106(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@39ea6e37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@a687359 -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490))) after lookup
08:20:50.857 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:50.858 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.859 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:50.859 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:51.117 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:51.117 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:51.118 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=9, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=379), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [15ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [15ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [59ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [62ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [62ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [118ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [119ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [121ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [121ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [373ms] refreshing cache from master, [373ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [378ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@112167106(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@39ea6e37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490) -> (continuation of Deferred@882866224 after retry RPC@853107588), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@a687359 -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490) -> (continuation of Deferred@882866224 after retry RPC@853107588))) after lookup
08:20:51.119 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:51.120 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:51.120 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:51.121 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:51.709219  5891 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:51.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:51.987323  5916 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f80bc8516214471ca7ea7896026da4da)
08:20:51.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:51.987957  5916 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:51.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:51.990553  5916 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:51.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:51.993598  5279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:20:51.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:51.998831  5279 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:20:52.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.004247  5808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "8d09d29cb0944901a7b67274723e6a6b" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" is_pre_election: true
08:20:52.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.005172  5808 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d09d29cb0944901a7b67274723e6a6b in term 4.
08:20:52.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.006884  5280 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127; no voters: f80bc8516214471ca7ea7896026da4da
08:20:52.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.007725  5916 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 5 FOLLOWER]: Leader pre-election won for term 6
08:20:52.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.008080  5916 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 5 FOLLOWER]: Starting leader election (detected failure of leader f80bc8516214471ca7ea7896026da4da)
08:20:52.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.008415  5916 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 5 FOLLOWER]: Advancing to term 6
08:20:52.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.013090  5916 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:52.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.014709  5916 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 6 election: Requested vote from peers f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:52.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.015993  5808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "8d09d29cb0944901a7b67274723e6a6b" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:20:52.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.016582  5808 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 4 FOLLOWER]: Advancing to term 6
08:20:52.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:52.018747  5279 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:20:52.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.023794  5808 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d09d29cb0944901a7b67274723e6a6b in term 6.
08:20:52.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.025091  5280 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127; no voters: f80bc8516214471ca7ea7896026da4da
08:20:52.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.025763  5916 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 6 FOLLOWER]: Leader election won for term 6
08:20:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.027652  5916 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 6 LEADER]: Becoming Leader. State: Replica: 8d09d29cb0944901a7b67274723e6a6b, State: Running, Role: LEADER
08:20:52.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.029215  5916 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:52.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.039580  3477 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b reported cstate change: term changed from 5 to 6, leader changed from f80bc8516214471ca7ea7896026da4da (127.3.12.129) to 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131). New cstate: current_term: 6 leader_uuid: "8d09d29cb0944901a7b67274723e6a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: UNKNOWN } } }
08:20:52.056 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:52.057 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:20:52.058 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=11, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1319), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [15ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [15ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [59ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [62ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [62ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [118ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [119ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [121ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [121ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [373ms] refreshing cache from master, [373ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [378ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [380ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [382ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [383ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [1313ms] refreshing cache from master, [1313ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1318ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@112167106(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@39ea6e37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490) -> (continuation of Deferred@882866224 after retry RPC@853107588) -> (continuation of Deferred@1876218216 after retry RPC@464428453), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@a687359 -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490) -> (continuation of Deferred@882866224 after retry RPC@853107588) -> (continuation of Deferred@1876218216 after retry RPC@464428453))) after lookup
08:20:52.060 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:20:52.065 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:52.065 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:52.065 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:20:52.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.526685  5808 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 6 FOLLOWER]: Refusing update from remote peer 8d09d29cb0944901a7b67274723e6a6b: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
08:20:52.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:52.528781  5916 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.001s
08:20:52.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:52.573967  5279 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b -> Peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Couldn't send request to peer f80bc8516214471ca7ea7896026da4da. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:20:54.856 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:54.857 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:54.857 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=13, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=4119), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [0ms] delaying RPC due to: Illegal state: connection is terminated, [13ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [15ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [15ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [41ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [41ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [59ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [62ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [62ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [118ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [119ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [121ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [121ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [373ms] refreshing cache from master, [373ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [378ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [380ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [382ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [383ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [1313ms] refreshing cache from master, [1313ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1318ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [1321ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [1327ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [1327ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [4113ms] refreshing cache from master, [4113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [4118ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@112167106(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@39ea6e37 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490) -> (continuation of Deferred@882866224 after retry RPC@853107588) -> (continuation of Deferred@1876218216 after retry RPC@464428453) -> (continuation of Deferred@888823340 after retry RPC@468504885), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@a687359 -> wakeup thread Time-limited test -> (continuation of Deferred@1696575402 after retry RPC@749009643) -> (continuation of Deferred@1431557769 after retry RPC@1588272490) -> (continuation of Deferred@882866224 after retry RPC@853107588) -> (continuation of Deferred@1876218216 after retry RPC@464428453) -> (continuation of Deferred@888823340 after retry RPC@468504885))) after lookup
08:20:54.858 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:54.859 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] Successfully connected to peer
08:20:54.883 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0xed177311, L:/127.0.0.1:53456 - R:/127.3.12.131:42197] using TOKEN/null
08:20:54.940 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.129:46667
08:20:54.942 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:20:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:54.944406  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:46667
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38639
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:55.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.008513  5279 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b -> Peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Couldn't send request to peer f80bc8516214471ca7ea7896026da4da. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:20:55.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.287858  6015 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:55.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.288375  6015 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:55.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.288920  6015 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:55.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.325742  6015 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:55.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.326742  6015 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.129
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:55.364635  6015 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:46667
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38639
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:55.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:55.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:55.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:55.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:55.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:55.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:55.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:55.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:55.366122  6015 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:55.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:55.367898  6015 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:55.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.380649  6023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:56.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.385208  6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:56.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:55.381945  6024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:20:56.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.544083  6015 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:20:56.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:56.544044  6027 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
08:20:56.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.548166  6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:20:56.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.550532  6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:20:56.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.551980  6015 hybrid_clock.cc:648] HybridClock initialized: now 1786522856551944 us; error 43 us; skew 500 ppm
08:20:56.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.552914  6015 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:20:56.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.560299  6015 webserver.cc:533] Webserver started at http://127.3.12.129:38639/ using document root <none> and password file <none>
08:20:56.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.561801  6015 fs_manager.cc:362] Metadata directory not provided
08:20:56.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.562109  6015 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:20:56.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.571615  6015 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.003s
08:20:56.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.576802  6049 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:20:56.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.578214  6015 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
08:20:56.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.578608  6015 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:56.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:56.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:20 on dist-test-slave-t7g5"
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.580642  6015 fs_report.cc:389] FS layout report
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/data
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:20:56.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:20:56.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.643659  6015 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:20:56.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.645366  6015 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:20:56.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.645888  6015 kserver.cc:163] Server-wide thread pool size limit: 3276
08:20:56.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.648819  6015 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:20:56.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.656569  6067 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:20:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.665737  6015 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:20:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.666024  6015 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
08:20:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.666360  6015 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:20:56.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.672061  6015 ts_tablet_manager.cc:616] Registered 1 tablets
08:20:56.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.672286  6015 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.001s	sys 0.003s
08:20:56.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.672706  6067 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap starting.
08:20:56.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.734447  6067 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Log is configured to *not* fsync() on all Append() calls
08:20:56.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.815997  6067 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:20:56.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.817207  6067 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap complete.
08:20:56.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.819240  6067 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent bootstrapping tablet: real 0.147s	user 0.094s	sys 0.049s
08:20:56.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.846019  6067 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:56.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.849543  6067 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Initialized, Role: FOLLOWER
08:20:56.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.851083  6067 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:56.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.857494  6067 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent starting tablet: real 0.038s	user 0.033s	sys 0.004s
08:20:56.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.883456  6015 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.129:46667
08:20:56.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.884055  6291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.129:46667 every 8 connection(s)
08:20:56.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.886384  6015 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:20:56.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.892272  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 6015
08:20:56.893 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:56.927 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:20:56.928 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.131:42197
08:20:56.928 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:20:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.929199  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 5244
08:20:56.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.968901  6292 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:20:56.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.969570  6292 heartbeater.cc:461] Registering TS with master...
08:20:56.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.970105  6293 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:20:56.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.970480  6293 heartbeater.cc:461] Registering TS with master...
08:20:56.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.971663  6293 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:20:56.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.971295  6292 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:20:56.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.977171  3339 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:56.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.978880  3477 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:56.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.982717  3477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.129:44727
08:20:56.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.989099  6294 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:20:56.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.989514  6294 heartbeater.cc:461] Registering TS with master...
08:20:56.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.990228  6294 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:20:56.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:56.994930  3204 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:57.007 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] handling channelInactive
08:20:57.007 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state READY due to: connection closed
08:20:57.007 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:20:57.009 [DEBUG - Time-limited test] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:57.010 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.011 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.011 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.036 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:57.037 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:57.037 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [3ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [23ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [28ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@451519852(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@67f1ba7d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@511dc352 -> wakeup thread Time-limited test)) after lookup
08:20:57.038 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:57.039 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.040 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.040 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.057 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:57.057 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:57.058 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [3ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [23ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [28ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [29ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [32ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@451519852(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@67f1ba7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@511dc352 -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985))) after lookup
08:20:57.059 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:57.060 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.060 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.061 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.096 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:57.097 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:57.097 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=89), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [3ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [23ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [28ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [29ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [32ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [50ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [53ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [53ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [83ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [88ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@451519852(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@67f1ba7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@511dc352 -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570))) after lookup
08:20:57.098 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:57.100 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.100 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.100 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.216 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:57.217 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:57.217 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=209), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [3ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [23ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [28ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [29ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [32ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [50ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [53ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [53ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [83ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [88ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [90ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [92ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [92ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [203ms] refreshing cache from master, [203ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [208ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@451519852(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@67f1ba7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570) -> (continuation of Deferred@207417476 after retry RPC@619185260), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@511dc352 -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570) -> (continuation of Deferred@207417476 after retry RPC@619185260))) after lookup
08:20:57.218 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:57.219 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.219 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.219 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.517 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:57.517 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:57.517 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=509), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [3ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [23ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [28ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [29ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [32ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [50ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [53ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [53ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [83ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [88ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [90ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [92ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [92ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [203ms] refreshing cache from master, [203ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [208ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [209ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [211ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [211ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [503ms] refreshing cache from master, [503ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [508ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@451519852(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@67f1ba7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570) -> (continuation of Deferred@207417476 after retry RPC@619185260) -> (continuation of Deferred@1883420371 after retry RPC@351082451), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@511dc352 -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570) -> (continuation of Deferred@207417476 after retry RPC@619185260) -> (continuation of Deferred@1883420371 after retry RPC@351082451))) after lookup
08:20:57.518 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:20:57.519 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.519 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.520 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:20:57.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:57.987592  6292 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:20:58.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.041843  6313 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:20:58.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.042445  6313 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:58.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.044889  6313 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:20:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:58.049846  6050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:58.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:58.055809  6050 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:58.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.059304  5808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" is_pre_election: true
08:20:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.061056  6053 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127
08:20:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.061856  6313 raft_consensus.cc:2749] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
08:20:58.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.330582  6319 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8d09d29cb0944901a7b67274723e6a6b)
08:20:58.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.331049  6319 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:58.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.333600  6319 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:58.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:58.337024  5657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:58.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:58.341600  5657 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:58.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.348100  6201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da" is_pre_election: true
08:20:58.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.348819  6201 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 5.
08:20:58.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.350606  5657 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b
08:20:58.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.351635  6319 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 6 FOLLOWER]: Leader pre-election won for term 7
08:20:58.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.352018  6319 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8d09d29cb0944901a7b67274723e6a6b)
08:20:58.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.352325  6319 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 6 FOLLOWER]: Advancing to term 7
08:20:58.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.358076  6319 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:58.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.359997  6319 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 7 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:20:58.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.362130  6201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da"
08:20:58.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.362767  6201 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 5 FOLLOWER]: Advancing to term 7
08:20:58.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:58.364534  5657 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:20:58.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.371562  6201 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 7.
08:20:58.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.372928  5657 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b
08:20:58.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.373754  6319 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 7 FOLLOWER]: Leader election won for term 7
08:20:58.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.376219  6319 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 7 LEADER]: Becoming Leader. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Running, Role: LEADER
08:20:58.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.377333  6319 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:20:58.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.386606  3477 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 reported cstate change: term changed from 6 to 7, leader changed from 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131) to ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130). New cstate: current_term: 7 leader_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: HEALTHY } } }
08:20:58.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:58.768368  5657 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 -> Peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Couldn't send request to peer 8d09d29cb0944901a7b67274723e6a6b. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:20:58.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.967438  6201 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 7 FOLLOWER]: Refusing update from remote peer ea5f3bbdb1704cba8076e499c9ed1127: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
08:20:58.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:58.969560  6319 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.001s
08:20:59.417 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:20:59.418 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:20:59.418 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2410), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [3ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [23ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [28ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [29ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [32ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [50ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [53ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [53ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [83ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [88ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [90ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [92ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [92ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [203ms] refreshing cache from master, [203ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [208ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [209ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [211ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [211ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [503ms] refreshing cache from master, [503ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [508ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [510ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [512ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [512ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2403ms] refreshing cache from master, [2403ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [2409ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@451519852(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@67f1ba7d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570) -> (continuation of Deferred@207417476 after retry RPC@619185260) -> (continuation of Deferred@1883420371 after retry RPC@351082451) -> (continuation of Deferred@994772609 after retry RPC@1466465392), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@511dc352 -> wakeup thread Time-limited test -> (continuation of Deferred@878911492 after retry RPC@198926985) -> (continuation of Deferred@1498142592 after retry RPC@156514570) -> (continuation of Deferred@207417476 after retry RPC@619185260) -> (continuation of Deferred@1883420371 after retry RPC@351082451) -> (continuation of Deferred@994772609 after retry RPC@1466465392))) after lookup
08:20:59.419 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:20:59.421 [DEBUG - kudu-nio-1] (Connection.java:757) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] Successfully connected to peer
08:20:59.445 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xb72df7dd, L:/127.0.0.1:53878 - R:/127.3.12.130:37191] using TOKEN/null
08:20:59.499 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.131:42197
08:20:59.500 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:20:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:59.501878  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:20:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:20:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:20:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:20:59.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:20:59.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:42197
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42909
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:59.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:20:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:59.879081  6377 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:20:59.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:59.879667  6377 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:20:59.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:59.880244  6377 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:20:59.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:59.916577  6377 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:20:59.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:59.917476  6377 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.131
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:59.957556  6377 tablet_server_runner.cc:78] Tablet server non-default flags:
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:42197
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42909
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:20:59.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:20:59.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:20:59.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:20:59.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:59.958997  6377 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:20:59.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:20:59.960915  6377 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:20:59.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:59.976015  6389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:01.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:01.293365  5657 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 -> Peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Couldn't send request to peer 8d09d29cb0944901a7b67274723e6a6b. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:21:01.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:20:59.976130  6390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:01.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:01.349593  6395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:01.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:01.352754  6391 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1369 milliseconds
08:21:01.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:01.352841  6377 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.377s	user 0.528s	sys 0.812s
08:21:01.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:01.353226  6377 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.378s	user 0.528s	sys 0.812s
08:21:01.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.353556  6377 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:21:01.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.355226  6377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:21:01.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.358141  6377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:21:01.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.359640  6377 hybrid_clock.cc:648] HybridClock initialized: now 1786522861359505 us; error 131 us; skew 500 ppm
08:21:01.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.360857  6377 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:21:01.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.370640  6377 webserver.cc:533] Webserver started at http://127.3.12.131:42909/ using document root <none> and password file <none>
08:21:01.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.372145  6377 fs_manager.cc:362] Metadata directory not provided
08:21:01.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.372490  6377 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:21:01.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.384433  6377 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.005s	sys 0.005s
08:21:01.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.390921  6416 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:21:01.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.392596  6377 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.000s	sys 0.005s
08:21:01.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.392920  6377 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:01.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:21:01.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:26 on dist-test-slave-t7g5"
08:21:01.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.395174  6377 fs_report.cc:389] FS layout report
08:21:01.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:21:01.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:01.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/data
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:21:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:21:01.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.468097  6377 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:21:01.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.469847  6377 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:21:01.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.470399  6377 kserver.cc:163] Server-wide thread pool size limit: 3276
08:21:01.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.473305  6377 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:21:01.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.479615  6437 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:21:01.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.488500  6377 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:21:01.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.488799  6377 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
08:21:01.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.489157  6377 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:21:01.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.495951  6377 ts_tablet_manager.cc:616] Registered 1 tablets
08:21:01.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.496243  6377 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.005s	sys 0.000s
08:21:01.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.496809  6437 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap starting.
08:21:01.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.566396  6437 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Log is configured to *not* fsync() on all Append() calls
08:21:01.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.664516  6437 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:21:01.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.665704  6437 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap complete.
08:21:01.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.667018  6437 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent bootstrapping tablet: real 0.171s	user 0.140s	sys 0.028s
08:21:01.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.689738  6377 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.131:42197
08:21:01.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.689873  6637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.131:42197 every 8 connection(s)
08:21:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.688441  6437 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.691967  6437 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09d29cb0944901a7b67274723e6a6b, State: Initialized, Role: FOLLOWER
08:21:01.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.693768  6377 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:21:01.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.693403  6437 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:01.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.698351  6437 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent starting tablet: real 0.031s	user 0.023s	sys 0.008s
08:21:01.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.704293  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 6377
08:21:01.706 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:01.743 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:21:01.744 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.130:37191
08:21:01.744 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:21:01.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.745317  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 5614
08:21:01.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.782145  6640 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:21:01.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.782727  6640 heartbeater.cc:461] Registering TS with master...
08:21:01.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.784533  6640 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:21:01.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.787779  6643 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:21:01.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.788182  6643 heartbeater.cc:461] Registering TS with master...
08:21:01.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.789178  6643 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:21:01.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.790666  3478 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:21:01.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.792572  3339 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:21:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.793509  3478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.131:51887
08:21:01.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.799250  6640 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:21:01.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.804682  6644 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:21:01.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.805087  6644 heartbeater.cc:461] Registering TS with master...
08:21:01.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.805790  6644 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:21:01.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:01.809370  3203 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:21:01.819 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:01.819 [DEBUG - kudu-nio-1] (Connection.java:239) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] handling channelInactive
08:21:01.820 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state READY due to: connection closed
08:21:01.821 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: connection closed
08:21:01.836 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:01.836 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:01.836 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@910248776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@22de2a0c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3be567d6 -> wakeup thread Time-limited test)) after lookup
08:21:01.837 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:01.838 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.838 [DEBUG - kudu-nio-2] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.838 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.856 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:01.856 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:01.856 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [18ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [18ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [36ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@910248776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@22de2a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3be567d6 -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889))) after lookup
08:21:01.857 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:01.858 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.858 [DEBUG - kudu-nio-3] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.858 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.897 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:01.897 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:01.898 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [18ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [18ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [36ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [37ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [38ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [38ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@910248776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@22de2a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3be567d6 -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782))) after lookup
08:21:01.899 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:01.900 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.900 [DEBUG - kudu-nio-4] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:01.900 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:02.037 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:02.037 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:02.038 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=218), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [18ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [18ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [36ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [37ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [38ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [38ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [78ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [80ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [80ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [217ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@910248776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@22de2a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782) -> (continuation of Deferred@1031107591 after retry RPC@1343811660), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3be567d6 -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782) -> (continuation of Deferred@1031107591 after retry RPC@1343811660))) after lookup
08:21:02.038 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:02.040 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:02.040 [DEBUG - kudu-nio-5] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:02.040 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:02.255 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:02.256 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:02.256 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=436), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [18ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [18ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [36ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [37ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [38ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [38ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [78ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [80ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [80ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [217ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [218ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [220ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [220ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [435ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@910248776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@22de2a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782) -> (continuation of Deferred@1031107591 after retry RPC@1343811660) -> (continuation of Deferred@821547899 after retry RPC@662359993), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3be567d6 -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782) -> (continuation of Deferred@1031107591 after retry RPC@1343811660) -> (continuation of Deferred@821547899 after retry RPC@662359993))) after lookup
08:21:02.257 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:02.258 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:02.258 [DEBUG - kudu-nio-6] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:02.259 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:02.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.791157  6660 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:21:02.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.791712  6660 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:02.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.793592  6660 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:02.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:02.803176  6053 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:21:02.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.810698  6564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b" is_pre_election: true
08:21:02.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.811686  6564 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 6.
08:21:02.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.813410  6050 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:21:02.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.814318  6660 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 7 FOLLOWER]: Leader pre-election won for term 8
08:21:02.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.814667  6660 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 7 FOLLOWER]: Starting leader election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:21:02.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.814993  6660 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 7 FOLLOWER]: Advancing to term 8
08:21:02.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.820958  6660 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:02.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.823222  6660 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 8 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:02.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.824165  6564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:21:02.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.824757  6564 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 6 FOLLOWER]: Advancing to term 8
08:21:02.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:02.827682  6053 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:21:02.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.833293  6564 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 8.
08:21:02.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.834900  6050 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:21:02.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.835758  6660 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 8 FOLLOWER]: Leader election won for term 8
08:21:02.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.837395  6660 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 8 LEADER]: Becoming Leader. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Running, Role: LEADER
08:21:02.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.839252  6660 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:02.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:02.848402  3478 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da reported cstate change: term changed from 7 to 8, leader changed from ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130) to f80bc8516214471ca7ea7896026da4da (127.3.12.129). New cstate: current_term: 8 leader_uuid: "f80bc8516214471ca7ea7896026da4da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: UNKNOWN } } }
08:21:03.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:03.235394  6564 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 8 FOLLOWER]: Refusing update from remote peer f80bc8516214471ca7ea7896026da4da: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
08:21:03.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:03.237389  6660 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.001s
08:21:03.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:03.267603  6053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111) [suppressed 2 similar messages]
08:21:03.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:03.273402  6053 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da -> Peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Couldn't send request to peer ea5f3bbdb1704cba8076e499c9ed1127. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:21:04.296 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:04.296 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:04.297 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2477), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [1ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [17ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [18ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [18ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [31ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [36ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [37ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [38ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [38ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [78ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [80ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [80ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [217ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [218ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [220ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [220ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [435ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [436ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [439ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [439ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [2472ms] refreshing cache from master, [2472ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [2476ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@910248776(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@22de2a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782) -> (continuation of Deferred@1031107591 after retry RPC@1343811660) -> (continuation of Deferred@821547899 after retry RPC@662359993) -> (continuation of Deferred@1819606355 after retry RPC@486727779), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3be567d6 -> wakeup thread Time-limited test -> (continuation of Deferred@1753034135 after retry RPC@2025204889) -> (continuation of Deferred@2097158827 after retry RPC@874193782) -> (continuation of Deferred@1031107591 after retry RPC@1343811660) -> (continuation of Deferred@821547899 after retry RPC@662359993) -> (continuation of Deferred@1819606355 after retry RPC@486727779))) after lookup
08:21:04.298 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:04.299 [DEBUG - kudu-nio-7] (Connection.java:757) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] Successfully connected to peer
08:21:04.335 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x7dff552e, L:/127.0.0.1:47066 - R:/127.3.12.129:46667] using TOKEN/null
08:21:04.399 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.130:37191
08:21:04.399 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:04.400681  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:04.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:21:04.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:21:04.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:04.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:04.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:04.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:21:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:04.771539  6741 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:21:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:04.772308  6741 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:21:04.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:04.773001  6741 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:21:04.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:04.811512  6741 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:21:04.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:04.812409  6741 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.130
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:04.876355  6741 tablet_server_runner.cc:78] Tablet server non-default flags:
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:21:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:21:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:21:04.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:04.878840  6741 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:21:04.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:04.881837  6741 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:21:04.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:04.898538  6758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:05.674840  6053 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da -> Peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Couldn't send request to peer ea5f3bbdb1704cba8076e499c9ed1127. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:21:06.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:04.901134  6759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:06.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:06.208050  6741 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.308s	user 0.005s	sys 0.001s
08:21:06.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:06.208422  6741 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.309s	user 0.005s	sys 0.001s
08:21:06.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.209939  6741 server_base.cc:1061] running on GCE node
08:21:06.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:06.211625  6778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:06.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.213145  6741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:21:06.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.215981  6741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:21:06.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.217484  6741 hybrid_clock.cc:648] HybridClock initialized: now 1786522866217353 us; error 123 us; skew 500 ppm
08:21:06.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.218675  6741 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:21:06.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.228435  6741 webserver.cc:533] Webserver started at http://127.3.12.130:39405/ using document root <none> and password file <none>
08:21:06.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.229447  6741 fs_manager.cc:362] Metadata directory not provided
08:21:06.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.229707  6741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:21:06.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.238875  6741 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.004s	sys 0.003s
08:21:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.244848  6787 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:21:06.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.246104  6741 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
08:21:06.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.246415  6741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:06.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:21:06.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:23 on dist-test-slave-t7g5"
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.248661  6741 fs_report.cc:389] FS layout report
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/data
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:21:06.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:21:06.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.340411  6741 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:21:06.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.342029  6741 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:21:06.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.342483  6741 kserver.cc:163] Server-wide thread pool size limit: 3276
08:21:06.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.345355  6741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:21:06.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.351533  6796 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:21:06.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.361299  6741 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:21:06.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.361753  6741 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.000s
08:21:06.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.362147  6741 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:21:06.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.369062  6741 ts_tablet_manager.cc:616] Registered 1 tablets
08:21:06.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.369313  6741 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.001s	sys 0.003s
08:21:06.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.370127  6796 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap starting.
08:21:06.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.438714  6796 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Log is configured to *not* fsync() on all Append() calls
08:21:06.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.542977  6796 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:21:06.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.544950  6796 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap complete.
08:21:06.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.547034  6796 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent bootstrapping tablet: real 0.178s	user 0.096s	sys 0.076s
08:21:06.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.571051  6741 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.130:37191
08:21:06.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.568106  6796 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:06.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.571388  6796 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Initialized, Role: FOLLOWER
08:21:06.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.571477  7009 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.130:37191 every 8 connection(s)
08:21:06.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.572820  6796 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:06.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.574235  6741 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:21:06.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.578003  6796 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent starting tablet: real 0.031s	user 0.028s	sys 0.002s
08:21:06.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.580467  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 6741
08:21:06.581 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:06.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:06.629786  3528 debug-util.cc:398] Leaking SignalData structure 0x7b08000aabc0 after lost signal to thread 3418
08:21:06.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.662384  7015 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:21:06.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.663089  7015 heartbeater.cc:461] Registering TS with master...
08:21:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.664758  7015 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:21:06.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.676756  7016 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:21:06.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.677181  7016 heartbeater.cc:461] Registering TS with master...
08:21:06.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.677106  3477 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:06.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.678179  7016 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:21:06.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.685092  3477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.130:40825
08:21:06.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.687752  3340 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:06.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.690414  7015 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:21:06.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.702705  7018 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:21:06.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.703178  7018 heartbeater.cc:461] Registering TS with master...
08:21:06.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.703927  7018 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:21:06.718 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:21:06.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.710162  3203 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:06.722 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.129:46667
08:21:06.722 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:21:06.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:06.723234  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 6015
08:21:06.763 [DEBUG - kudu-nio-7] (Connection.java:239) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] handling channelInactive
08:21:06.763 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state READY due to: connection closed
08:21:06.764 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:06.765 [DEBUG - Time-limited test] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:06.766 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.766 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.766 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.776 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:06.777 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:06.777 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1047808405(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2a39823e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5a3cc80b -> wakeup thread Time-limited test)) after lookup
08:21:06.778 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:06.779 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.780 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.780 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.796 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:06.797 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:06.797 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [14ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [16ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [16ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1047808405(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2a39823e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5a3cc80b -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095))) after lookup
08:21:06.797 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:06.799 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.799 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.799 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.816 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:06.816 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:06.817 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [14ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [16ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [16ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [33ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [35ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [35ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1047808405(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2a39823e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5a3cc80b -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311))) after lookup
08:21:06.818 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:06.819 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.819 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.819 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.876 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:06.876 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:06.876 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=111), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [14ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [16ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [16ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [33ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [35ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [35ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [53ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [55ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [55ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1047808405(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2a39823e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5a3cc80b -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565))) after lookup
08:21:06.877 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:06.878 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.878 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:06.878 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:07.255 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:07.256 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:07.256 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=491), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [14ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [16ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [16ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [33ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [35ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [35ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [53ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [55ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [55ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [112ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [114ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [114ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [487ms] refreshing cache from master, [487ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [491ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1047808405(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2a39823e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565) -> (continuation of Deferred@1879835843 after retry RPC@965670729), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5a3cc80b -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565) -> (continuation of Deferred@1879835843 after retry RPC@965670729))) after lookup
08:21:07.257 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:07.258 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:07.259 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:07.259 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:07.476 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:07.476 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:07.476 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=712), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [14ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [16ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [16ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [33ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [35ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [35ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [53ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [55ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [55ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [112ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [114ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [114ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [487ms] refreshing cache from master, [487ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [491ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [493ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [495ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [495ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [707ms] refreshing cache from master, [707ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [712ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1047808405(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2a39823e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565) -> (continuation of Deferred@1879835843 after retry RPC@965670729) -> (continuation of Deferred@182544981 after retry RPC@1151066367), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5a3cc80b -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565) -> (continuation of Deferred@1879835843 after retry RPC@965670729) -> (continuation of Deferred@182544981 after retry RPC@1151066367))) after lookup
08:21:07.478 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:07.479 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:07.479 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state CONNECTING due to: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:07.479 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667
08:21:07.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.866634  7044 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f80bc8516214471ca7ea7896026da4da)
08:21:07.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.867091  7044 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:07.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.869565  7044 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:07.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:07.872924  6419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:21:07.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:07.889909  6419 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:21:07.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.895112  6919 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "8d09d29cb0944901a7b67274723e6a6b" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" is_pre_election: true
08:21:07.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.896041  6919 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d09d29cb0944901a7b67274723e6a6b in term 7.
08:21:07.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.897517  6420 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127; no voters: f80bc8516214471ca7ea7896026da4da
08:21:07.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.898320  7044 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 8 FOLLOWER]: Leader pre-election won for term 9
08:21:07.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.898751  7044 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 8 FOLLOWER]: Starting leader election (detected failure of leader f80bc8516214471ca7ea7896026da4da)
08:21:07.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.899010  7044 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 8 FOLLOWER]: Advancing to term 9
08:21:07.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.903628  7044 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:07.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.905155  7044 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 9 election: Requested vote from peers f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:07.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.906656  6919 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "8d09d29cb0944901a7b67274723e6a6b" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:21:07.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.907238  6919 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 7 FOLLOWER]: Advancing to term 9
08:21:07.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:07.909158  6419 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111)
08:21:07.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.917037  6919 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d09d29cb0944901a7b67274723e6a6b in term 9.
08:21:07.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.918571  6420 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, ea5f3bbdb1704cba8076e499c9ed1127; no voters: f80bc8516214471ca7ea7896026da4da
08:21:07.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.919543  7044 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 9 FOLLOWER]: Leader election won for term 9
08:21:07.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.921330  7044 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 9 LEADER]: Becoming Leader. State: Replica: 8d09d29cb0944901a7b67274723e6a6b, State: Running, Role: LEADER
08:21:07.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.923453  7044 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:07.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:07.939563  3477 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b reported cstate change: term changed from 8 to 9, leader changed from f80bc8516214471ca7ea7896026da4da (127.3.12.129) to 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131). New cstate: current_term: 9 leader_uuid: "8d09d29cb0944901a7b67274723e6a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: UNKNOWN } } }
08:21:08.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:08.544180  6919 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 9 FOLLOWER]: Refusing update from remote peer 8d09d29cb0944901a7b67274723e6a6b: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
08:21:08.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:08.545781  7070 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
08:21:08.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:08.554311  6419 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b -> Peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Couldn't send request to peer f80bc8516214471ca7ea7896026da4da. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:21:08.936 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:08.936 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:08.937 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=14, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2172), Traces: [0ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [2ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [14ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [16ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [16ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [32ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [33ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [35ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [35ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [52ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [53ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [55ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [55ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [112ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [114ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [114ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [487ms] refreshing cache from master, [487ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [491ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [493ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [495ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [495ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [707ms] refreshing cache from master, [707ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [712ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [713ms] sending RPC to server f80bc8516214471ca7ea7896026da4da, [715ms] received response from server f80bc8516214471ca7ea7896026da4da: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [715ms] delaying RPC due to: Network error: Failed to connect to peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667): Connection refused: /127.3.12.129:46667, [2167ms] refreshing cache from master, [2167ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [2172ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1047808405(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@2a39823e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565) -> (continuation of Deferred@1879835843 after retry RPC@965670729) -> (continuation of Deferred@182544981 after retry RPC@1151066367) -> (continuation of Deferred@1094739094 after retry RPC@1514661977), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5a3cc80b -> wakeup thread Time-limited test -> (continuation of Deferred@86650097 after retry RPC@971686095) -> (continuation of Deferred@1465487379 after retry RPC@945479311) -> (continuation of Deferred@1903876139 after retry RPC@523947565) -> (continuation of Deferred@1879835843 after retry RPC@965670729) -> (continuation of Deferred@182544981 after retry RPC@1151066367) -> (continuation of Deferred@1094739094 after retry RPC@1514661977))) after lookup
08:21:08.938 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:21:08.939 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] Successfully connected to peer
08:21:08.961 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0xb3f8105b, L:/127.0.0.1:39786 - R:/127.3.12.131:42197] using TOKEN/null
08:21:09.010 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.129:46667
08:21:09.011 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:09.012645  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:46667
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38639
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:21:09.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:09.366861  7097 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:21:09.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:09.367431  7097 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:21:09.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:09.368054  7097 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:21:09.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:09.406497  7097 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:21:09.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:09.407678  7097 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.129
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:09.449993  7097 tablet_server_runner.cc:78] Tablet server non-default flags:
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/logs
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.129:46667
08:21:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.129
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38639
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.129
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:21:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:21:09.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:09.451856  7097 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:21:09.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:09.453991  7097 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:21:09.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:09.470299  7114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:10.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:09.470659  7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:10.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:10.811707  7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:10.815094  7117 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1338 milliseconds
08:21:10.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:10.815814  7097 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.346s	user 0.452s	sys 0.829s
08:21:10.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:10.816075  7097 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.346s	user 0.452s	sys 0.829s
08:21:10.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.816275  7097 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:21:10.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.817395  7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:21:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.819670  7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:21:10.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.821208  7097 hybrid_clock.cc:648] HybridClock initialized: now 1786522870821136 us; error 50 us; skew 500 ppm
08:21:10.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.822379  7097 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:21:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.832258  7097 webserver.cc:533] Webserver started at http://127.3.12.129:38639/ using document root <none> and password file <none>
08:21:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.833402  7097 fs_manager.cc:362] Metadata directory not provided
08:21:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.833645  7097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:21:10.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.842356  7097 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.000s
08:21:10.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.847980  7155 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:21:10.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.849362  7097 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.003s
08:21:10.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.849754  7097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f80bc8516214471ca7ea7896026da4da"
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:20 on dist-test-slave-t7g5"
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.851948  7097 fs_report.cc:389] FS layout report
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/wal
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/data
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:21:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:21:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.955858  7097 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:21:10.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.957944  7097 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:21:10.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.958539  7097 kserver.cc:163] Server-wide thread pool size limit: 3276
08:21:10.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.962334  7097 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:21:10.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.970741  7165 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:21:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.980213  7097 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:21:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.980502  7097 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
08:21:10.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.980754  7097 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:21:10.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.985960  7097 ts_tablet_manager.cc:616] Registered 1 tablets
08:21:10.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.986182  7097 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.006s	sys 0.000s
08:21:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:10.986925  7165 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap starting.
08:21:11.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.072232  7165 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Log is configured to *not* fsync() on all Append() calls
08:21:11.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:11.118916  6419 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b -> Peer f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667): Couldn't send request to peer f80bc8516214471ca7ea7896026da4da. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.129:46667: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:21:11.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.179183  7097 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.129:46667
08:21:11.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.179325  7388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.129:46667 every 8 connection(s)
08:21:11.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.182852  7097 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-0/data/info.pb
08:21:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.185155  7165 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:21:11.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.186353  7165 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Bootstrap complete.
08:21:11.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.188508  7165 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent bootstrapping tablet: real 0.202s	user 0.153s	sys 0.043s
08:21:11.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.192593  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 7097
08:21:11.193 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:11.226 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:21:11.230 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.131:42197
08:21:11.230 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:21:11.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.230907  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 6377
08:21:11.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.249264  7165 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:11.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.254707  7165 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Initialized, Role: FOLLOWER
08:21:11.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.256168  7165 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:11.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.263101  7165 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da: Time spent starting tablet: real 0.074s	user 0.032s	sys 0.007s
08:21:11.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.268401  7389 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:21:11.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.268921  7389 heartbeater.cc:461] Registering TS with master...
08:21:11.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.270316  7389 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:21:11.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.275413  3478 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:21:11.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.278268  3478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.129:47035
08:21:11.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.282483  7393 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:21:11.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.282862  7393 heartbeater.cc:461] Registering TS with master...
08:21:11.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.283316  7390 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:21:11.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.283561  7393 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:21:11.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.283677  7390 heartbeater.cc:461] Registering TS with master...
08:21:11.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.284349  7390 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:21:11.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.284418  7389 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:21:11.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.287174  3203 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:21:11.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:11.287807  3340 ts_manager.cc:194] Re-registered known tserver with Master: f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:21:11.297 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] handling channelInactive
08:21:11.298 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state READY due to: connection closed
08:21:11.298 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:11.299 [DEBUG - Time-limited test] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:21:11.300 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.300 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.300 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.315 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:11.316 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:11.316 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1244524083(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@11482de7 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d4f1ed8 -> wakeup thread Time-limited test)) after lookup
08:21:11.316 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:21:11.317 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.318 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.318 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.336 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:11.336 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:11.337 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [20ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [20ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1244524083(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@11482de7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d4f1ed8 -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855))) after lookup
08:21:11.337 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:21:11.338 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.338 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.338 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.376 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:11.376 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:11.376 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [20ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [20ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [40ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [41ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1244524083(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@11482de7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d4f1ed8 -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027))) after lookup
08:21:11.377 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:21:11.378 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.378 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.378 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.416 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:11.416 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:11.416 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=118), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [20ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [20ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [40ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [41ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [78ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [80ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [81ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [117ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1244524083(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@11482de7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027) -> (continuation of Deferred@569125596 after retry RPC@522925819), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d4f1ed8 -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027) -> (continuation of Deferred@569125596 after retry RPC@522925819))) after lookup
08:21:11.417 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:21:11.418 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.418 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.418 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.715 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:11.715 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)[L],ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:11.715 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=417), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [20ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [20ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [40ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [41ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [78ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [80ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [81ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [117ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [118ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [120ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [120ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [413ms] refreshing cache from master, [413ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [417ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1244524083(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@11482de7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027) -> (continuation of Deferred@569125596 after retry RPC@522925819) -> (continuation of Deferred@962193474 after retry RPC@2139327868), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d4f1ed8 -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027) -> (continuation of Deferred@569125596 after retry RPC@522925819) -> (continuation of Deferred@962193474 after retry RPC@2139327868))) after lookup
08:21:11.716 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] connecting to peer
08:21:11.717 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.718 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:11.718 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197
08:21:12.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.164113  7415 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8d09d29cb0944901a7b67274723e6a6b)
08:21:12.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.164638  7415 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:12.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.167116  7415 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:21:12.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:12.170485  6788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:21:12.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:12.175014  6788 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:21:12.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.180935  7282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da" is_pre_election: true
08:21:12.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.181962  7282 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 8.
08:21:12.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.183660  6788 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b
08:21:12.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.184664  7415 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 9 FOLLOWER]: Leader pre-election won for term 10
08:21:12.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.185021  7415 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8d09d29cb0944901a7b67274723e6a6b)
08:21:12.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.185406  7415 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 9 FOLLOWER]: Advancing to term 10
08:21:12.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.192982  7415 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:12.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.195148  7415 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 10 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), f80bc8516214471ca7ea7896026da4da (127.3.12.129:46667)
08:21:12.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.198377  7282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "f80bc8516214471ca7ea7896026da4da"
08:21:12.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.199043  7282 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 8 FOLLOWER]: Advancing to term 10
08:21:12.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:12.200284  6788 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111)
08:21:12.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.207723  7282 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5f3bbdb1704cba8076e499c9ed1127 in term 10.
08:21:12.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.209084  6788 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea5f3bbdb1704cba8076e499c9ed1127, f80bc8516214471ca7ea7896026da4da; no voters: 8d09d29cb0944901a7b67274723e6a6b
08:21:12.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.210067  7415 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 10 FOLLOWER]: Leader election won for term 10
08:21:12.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.212337  7415 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 10 LEADER]: Becoming Leader. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Running, Role: LEADER
08:21:12.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.213359  7415 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:12.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.225332  3478 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 reported cstate change: term changed from 9 to 10, leader changed from 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131) to ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130). New cstate: current_term: 10 leader_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: HEALTHY } } }
08:21:12.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.748956  7282 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 10 FOLLOWER]: Refusing update from remote peer ea5f3bbdb1704cba8076e499c9ed1127: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
08:21:12.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:12.750746  7455 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
08:21:12.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:12.757979  6788 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 -> Peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Couldn't send request to peer 8d09d29cb0944901a7b67274723e6a6b. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:21:13.556 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:13.557 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:13.557 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2258), Traces: [0ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [2ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [18ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [20ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [20ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [39ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [40ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [41ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [77ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [78ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [80ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [81ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [113ms] refreshing cache from master, [113ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [117ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [118ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [120ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [120ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [413ms] refreshing cache from master, [413ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [417ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [418ms] sending RPC to server 8d09d29cb0944901a7b67274723e6a6b, [420ms] received response from server 8d09d29cb0944901a7b67274723e6a6b: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [420ms] delaying RPC due to: Network error: Failed to connect to peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197): Connection refused: /127.3.12.131:42197, [2253ms] refreshing cache from master, [2254ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [2258ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1244524083(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@11482de7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027) -> (continuation of Deferred@569125596 after retry RPC@522925819) -> (continuation of Deferred@962193474 after retry RPC@2139327868) -> (continuation of Deferred@1210397358 after retry RPC@724479675), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d4f1ed8 -> wakeup thread Time-limited test -> (continuation of Deferred@10131442 after retry RPC@1172762855) -> (continuation of Deferred@2108006240 after retry RPC@255212027) -> (continuation of Deferred@569125596 after retry RPC@522925819) -> (continuation of Deferred@962193474 after retry RPC@2139327868) -> (continuation of Deferred@1210397358 after retry RPC@724479675))) after lookup
08:21:13.558 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:13.559 [DEBUG - kudu-nio-6] (Connection.java:757) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] Successfully connected to peer
08:21:13.594 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0xcd83da83, L:/127.0.0.1:53230 - R:/127.3.12.130:37191] using TOKEN/null
08:21:13.650 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.131:42197
08:21:13.650 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:21:13.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:13.652405  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:42197
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42909
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:21:14.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:14.023972  7486 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:21:14.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:14.024664  7486 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:21:14.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:14.025180  7486 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:21:14.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:14.062517  7486 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:21:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:14.063537  7486 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.131
08:21:14.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:14.108399  7486 tablet_server_runner.cc:78] Tablet server non-default flags:
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/logs
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:14.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.131:42197
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.131
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42909
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.131
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:21:14.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:21:14.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:14.110697  7486 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:21:14.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:14.113333  7486 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:21:14.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:14.128880  7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:15.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:15.144428  6788 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 -> Peer 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197): Couldn't send request to peer 8d09d29cb0944901a7b67274723e6a6b. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.131:42197: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:21:15.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:15.533634  7496 debug-util.cc:398] Leaking SignalData structure 0x7b080002ee40 after lost signal to thread 7486
08:21:15.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:15.706044  7486 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.576s	user 0.674s	sys 0.889s
08:21:15.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:14.130386  7502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:15.706764  7486 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.578s	user 0.675s	sys 0.890s
08:21:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:15.708616  7504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:15.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:15.711694  7503 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1578 milliseconds
08:21:15.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.711768  7486 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:21:15.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.713032  7486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:21:15.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.715224  7486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:21:15.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.716605  7486 hybrid_clock.cc:648] HybridClock initialized: now 1786522875716578 us; error 58 us; skew 500 ppm
08:21:15.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.717483  7486 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:21:15.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.724889  7486 webserver.cc:533] Webserver started at http://127.3.12.131:42909/ using document root <none> and password file <none>
08:21:15.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.725999  7486 fs_manager.cc:362] Metadata directory not provided
08:21:15.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.726222  7486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:21:15.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.736172  7486 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.005s
08:21:15.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.741298  7523 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:21:15.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.742929  7486 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.003s
08:21:15.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.743433  7486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:26 on dist-test-slave-t7g5"
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.745846  7486 fs_report.cc:389] FS layout report
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/wal
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/data
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:21:15.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:21:15.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:21:15.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.801084  7486 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:21:15.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.802908  7486 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:21:15.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.803422  7486 kserver.cc:163] Server-wide thread pool size limit: 3276
08:21:15.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.806468  7486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:21:15.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.814239  7543 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:21:15.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.824010  7486 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:21:15.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.824347  7486 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.000s	sys 0.003s
08:21:15.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.824690  7486 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:21:15.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.830268  7486 ts_tablet_manager.cc:616] Registered 1 tablets
08:21:15.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.830490  7486 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.002s	sys 0.003s
08:21:15.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.831012  7543 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap starting.
08:21:15.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:15.900494  7543 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Log is configured to *not* fsync() on all Append() calls
08:21:16.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.015678  7543 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:21:16.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.016780  7543 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Bootstrap complete.
08:21:16.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.018325  7543 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent bootstrapping tablet: real 0.188s	user 0.134s	sys 0.051s
08:21:16.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.021754  7486 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.131:42197
08:21:16.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.021916  7737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.131:42197 every 8 connection(s)
08:21:16.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.024528  7486 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-2/data/info.pb
08:21:16.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.028573  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 7486
08:21:16.029 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:16.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.036823  7543 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:16.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.040084  7543 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09d29cb0944901a7b67274723e6a6b, State: Initialized, Role: FOLLOWER
08:21:16.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.041252  7543 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:16.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.051766  7543 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b: Time spent starting tablet: real 0.033s	user 0.026s	sys 0.005s
08:21:16.070 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:21:16.071 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.12.130:37191
08:21:16.072 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:21:16.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.072420  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 6741
08:21:16.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.094569  7738 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:21:16.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.095168  7738 heartbeater.cc:461] Registering TS with master...
08:21:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.096712  7738 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:21:16.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.101225  3478 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:21:16.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.104372  3478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.131:56561
08:21:16.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.109117  7738 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:21:16.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.127660  7741 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:21:16.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.128058  7741 heartbeater.cc:461] Registering TS with master...
08:21:16.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.128651  7741 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:21:16.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.129501  7748 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:21:16.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.130036  7748 heartbeater.cc:461] Registering TS with master...
08:21:16.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.131942  3339 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:21:16.134 [DEBUG - kudu-nio-6] (Connection.java:239) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] handling channelInactive
08:21:16.134 [DEBUG - kudu-nio-6] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state READY due to: connection closed
08:21:16.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.135150  7748 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:21:16.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:16.138350  3203 ts_manager.cc:194] Re-registered known tserver with Master: 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197)
08:21:16.145 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:16.145 [DEBUG - Time-limited test] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:16.147 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.147 [DEBUG - kudu-nio-7] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.148 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.157 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:16.157 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:16.158 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1202422719(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@19aeb687 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@261892bc -> wakeup thread Time-limited test)) after lookup
08:21:16.158 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:16.159 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.160 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.160 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.175 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:16.176 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:16.176 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [15ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [15ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1202422719(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@19aeb687 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@261892bc -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989))) after lookup
08:21:16.177 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:16.178 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.178 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.178 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.214 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:16.215 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:16.215 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [15ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [15ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [31ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [33ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [33ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1202422719(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@19aeb687 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@261892bc -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341))) after lookup
08:21:16.216 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:16.217 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.217 [DEBUG - kudu-nio-2] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.217 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.356 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:16.356 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:16.356 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=211), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [15ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [15ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [31ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [33ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [33ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [70ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [72ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [73ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [206ms] refreshing cache from master, [206ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [210ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1202422719(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@19aeb687 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341) -> (continuation of Deferred@427461022 after retry RPC@1928257470), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@261892bc -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341) -> (continuation of Deferred@427461022 after retry RPC@1928257470))) after lookup
08:21:16.357 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:16.358 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.358 [DEBUG - kudu-nio-3] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.358 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.435 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:16.435 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)[L],f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)]}]
08:21:16.436 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=290), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [15ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [15ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [31ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [33ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [33ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [70ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [72ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [73ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [206ms] refreshing cache from master, [206ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [210ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [211ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [213ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [213ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [286ms] refreshing cache from master, [287ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [290ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1202422719(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@19aeb687 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341) -> (continuation of Deferred@427461022 after retry RPC@1928257470) -> (continuation of Deferred@1531666436 after retry RPC@1608275964), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@261892bc -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341) -> (continuation of Deferred@427461022 after retry RPC@1928257470) -> (continuation of Deferred@1531666436 after retry RPC@1608275964))) after lookup
08:21:16.436 [DEBUG - kudu-nio-0] (Connection.java:748) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] connecting to peer
08:21:16.437 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.438 [DEBUG - kudu-nio-4] (Connection.java:709) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] cleaning up while in state CONNECTING due to: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:16.438 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191) for tablet 203fe5a2ea01428e964d956caebbbf3c: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191
08:21:17.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.101094  7773 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:21:17.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.101572  7773 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:17.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.104074  7773 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:17.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:17.108481  7160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:21:17.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:17.114205  7160 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:21:17.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.118994  7652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b" is_pre_election: true
08:21:17.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.119855  7652 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 9.
08:21:17.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.121521  7159 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:21:17.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.122570  7773 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 10 FOLLOWER]: Leader pre-election won for term 11
08:21:17.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.122843  7773 raft_consensus.cc:493] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 10 FOLLOWER]: Starting leader election (detected failure of leader ea5f3bbdb1704cba8076e499c9ed1127)
08:21:17.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.123063  7773 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 10 FOLLOWER]: Advancing to term 11
08:21:17.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.127810  7773 raft_consensus.cc:515] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:17.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.129616  7773 leader_election.cc:290] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 11 election: Requested vote from peers 8d09d29cb0944901a7b67274723e6a6b (127.3.12.131:42197), ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:17.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.131224  7652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203fe5a2ea01428e964d956caebbbf3c" candidate_uuid: "f80bc8516214471ca7ea7896026da4da" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "8d09d29cb0944901a7b67274723e6a6b"
08:21:17.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.131806  7652 raft_consensus.cc:3060] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 9 FOLLOWER]: Advancing to term 11
08:21:17.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:17.134569  7160 leader_election.cc:336] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111)
08:21:17.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.139901  7652 raft_consensus.cc:2468] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f80bc8516214471ca7ea7896026da4da in term 11.
08:21:17.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.141016  7159 leader_election.cc:304] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d09d29cb0944901a7b67274723e6a6b, f80bc8516214471ca7ea7896026da4da; no voters: ea5f3bbdb1704cba8076e499c9ed1127
08:21:17.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.141738  7773 raft_consensus.cc:2804] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 11 FOLLOWER]: Leader election won for term 11
08:21:17.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.143718  7773 raft_consensus.cc:697] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [term 11 LEADER]: Becoming Leader. State: Replica: f80bc8516214471ca7ea7896026da4da, State: Running, Role: LEADER
08:21:17.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.144647  7773 consensus_queue.cc:237] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:17.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.154732  3478 catalog_manager.cc:5719] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da reported cstate change: term changed from 10 to 11, leader changed from ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130) to f80bc8516214471ca7ea7896026da4da (127.3.12.129). New cstate: current_term: 11 leader_uuid: "f80bc8516214471ca7ea7896026da4da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } health_report { overall_health: UNKNOWN } } }
08:21:17.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.633924  7652 raft_consensus.cc:1275] T 203fe5a2ea01428e964d956caebbbf3c P 8d09d29cb0944901a7b67274723e6a6b [term 11 FOLLOWER]: Refusing update from remote peer f80bc8516214471ca7ea7896026da4da: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
08:21:17.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.636013  7773 consensus_queue.cc:1048] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
08:21:17.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:17.668944  7160 consensus_peers.cc:597] T 203fe5a2ea01428e964d956caebbbf3c P f80bc8516214471ca7ea7896026da4da -> Peer ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191): Couldn't send request to peer ea5f3bbdb1704cba8076e499c9ed1127. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.130:37191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:21:17.856 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 203fe5a2ea01428e964d956caebbbf3c for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097' with partition [<start>, <end>)
08:21:17.857 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=203fe5a2ea01428e964d956caebbbf3c@[8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197),ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191),f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)[L]]}]
08:21:17.857 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=203fe5a2ea01428e964d956caebbbf3c, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1711), Traces: [0ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [2ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [3ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [12ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [13ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [15ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [15ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [31ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [33ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [33ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [70ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [72ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [73ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [206ms] refreshing cache from master, [206ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [210ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [211ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [213ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [213ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [286ms] refreshing cache from master, [287ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [290ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, [291ms] sending RPC to server ea5f3bbdb1704cba8076e499c9ed1127, [293ms] received response from server ea5f3bbdb1704cba8076e499c9ed1127: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [293ms] delaying RPC due to: Network error: Failed to connect to peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191): Connection refused: /127.3.12.130:37191, [1706ms] refreshing cache from master, [1706ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.12.188:42903, [1711ms] Sub RPC GetTableLocations: received response from server master-127.3.12.188:42903: OK, deferred=Deferred@1202422719(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007f7cc43391e8@19aeb687 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341) -> (continuation of Deferred@427461022 after retry RPC@1928257470) -> (continuation of Deferred@1531666436 after retry RPC@1608275964) -> (continuation of Deferred@1073997759 after retry RPC@2103262196), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@261892bc -> wakeup thread Time-limited test -> (continuation of Deferred@324065695 after retry RPC@776138989) -> (continuation of Deferred@571628306 after retry RPC@460194341) -> (continuation of Deferred@427461022 after retry RPC@1928257470) -> (continuation of Deferred@1531666436 after retry RPC@1608275964) -> (continuation of Deferred@1073997759 after retry RPC@2103262196))) after lookup
08:21:17.858 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] connecting to peer
08:21:17.860 [DEBUG - kudu-nio-5] (Connection.java:757) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] Successfully connected to peer
08:21:17.898 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0xfffa08f1, L:/127.0.0.1:60608 - R:/127.3.12.129:46667] using TOKEN/null
08:21:17.956 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.12.130:37191
08:21:17.956 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:17.957644  3122 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:17.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:21:18.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:18.326833  7822 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:21:18.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:18.327342  7822 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:21:18.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:18.327812  7822 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:21:18.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:18.364333  7822 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:21:18.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:18.365419  7822 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.130
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:18.406008  7822 tablet_server_runner.cc:78] Tablet server non-default flags:
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/logs
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.12.148:41869
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:21:18.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.12.130:37191
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.12.130
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39405
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.12.190:33597,127.3.12.189:34545,127.3.12.188:42903
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.12.130
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:20 UTC on e7f111948823
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13303
08:21:18.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:21:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:18.407773  7822 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:21:18.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:18.409600  7822 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:21:18.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:18.424875  7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:19.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:18.425966  7837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:19.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:19.708887  7838 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1279 milliseconds
08:21:19.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:19.710618  7842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:21:19.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:19.717013  7822 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.290s	user 0.000s	sys 0.006s
08:21:19.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:21:19.717446  7822 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.291s	user 0.000s	sys 0.006s
08:21:19.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.717702  7822 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:21:19.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.719041  7822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:21:19.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.721583  7822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:21:19.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.723031  7822 hybrid_clock.cc:648] HybridClock initialized: now 1786522879722981 us; error 61 us; skew 500 ppm
08:21:19.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.723994  7822 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:21:19.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.732802  7822 webserver.cc:533] Webserver started at http://127.3.12.130:39405/ using document root <none> and password file <none>
08:21:19.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.733892  7822 fs_manager.cc:362] Metadata directory not provided
08:21:19.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.734148  7822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:21:19.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.743319  7822 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.003s
08:21:19.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.749099  7857 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:21:19.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.750489  7822 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:21:19.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.750839  7822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data,/tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:19.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ea5f3bbdb1704cba8076e499c9ed1127"
08:21:19.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:20:23 on dist-test-slave-t7g5"
08:21:19.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.752885  7822 fs_report.cc:389] FS layout report
08:21:19.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:21:19.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/wal
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/data
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:21:19.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:21:19.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.836642  7822 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:21:19.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.838471  7822 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:21:19.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.838941  7822 kserver.cc:163] Server-wide thread pool size limit: 3276
08:21:19.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.841763  7822 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:21:19.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.848315  7866 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:21:19.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.857414  7822 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:21:19.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.857790  7822 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.003s
08:21:19.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.858052  7822 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:21:19.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.863384  7822 ts_tablet_manager.cc:616] Registered 1 tablets
08:21:19.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.863595  7822 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.000s	sys 0.004s
08:21:19.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.864174  7866 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap starting.
08:21:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:19.942780  7866 log.cc:826] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Log is configured to *not* fsync() on all Append() calls
08:21:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.064924  7822 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.130:37191
08:21:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.065079  8063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.12.130:37191 every 8 connection(s)
08:21:20.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.068174  7822 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskjuwuwQ/test-tmp/mini-kudu-cluster16881142711432121317/ts-1/data/info.pb
08:21:20.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.072943  3122 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu as pid 7822
08:21:20.073 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:21:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.118459  7866 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:21:20.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.119637  7866 tablet_bootstrap.cc:492] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Bootstrap complete.
08:21:20.120 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:21:20.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.121470  7866 ts_tablet_manager.cc:1403] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent bootstrapping tablet: real 0.258s	user 0.188s	sys 0.033s
08:21:20.138 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522827097, tablet=203fe5a2ea01428e964d956caebbbf3c, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:21:20.144 [DEBUG - main] (Connection.java:575) [peer master-127.3.12.188:42903(127.3.12.188:42903)] disconnecting while in state READY
08:21:20.145 [DEBUG - main] (Connection.java:575) [peer master-127.3.12.190:33597(127.3.12.190:33597)] disconnecting while in state READY
08:21:20.147 [DEBUG - main] (Connection.java:575) [peer ea5f3bbdb1704cba8076e499c9ed1127(127.3.12.130:37191)] disconnecting while in state TERMINATED
08:21:20.147 [DEBUG - main] (Connection.java:575) [peer 8d09d29cb0944901a7b67274723e6a6b(127.3.12.131:42197)] disconnecting while in state TERMINATED
08:21:20.147 [DEBUG - main] (Connection.java:575) [peer master-127.3.12.189:34545(127.3.12.189:34545)] disconnecting while in state READY
08:21:20.148 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.3.12.190:33597(127.3.12.190:33597)] handling channelInactive
08:21:20.148 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.3.12.188:42903(127.3.12.188:42903)] handling channelInactive
08:21:20.148 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.3.12.190:33597(127.3.12.190:33597)] cleaning up while in state READY due to: connection closed
08:21:20.148 [DEBUG - main] (Connection.java:575) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] disconnecting while in state READY
08:21:20.149 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.3.12.188:42903(127.3.12.188:42903)] cleaning up while in state READY due to: connection closed
08:21:20.154 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.3.12.189:34545(127.3.12.189:34545)] handling channelInactive
08:21:20.154 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.3.12.189:34545(127.3.12.189:34545)] cleaning up while in state READY due to: connection closed
08:21:20.156 [DEBUG - kudu-nio-5] (AsyncKuduClient.java:2704) Releasing all remaining resources
08:21:20.158 [DEBUG - kudu-nio-5] (Connection.java:239) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] handling channelInactive
08:21:20.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.158476  8065 heartbeater.cc:344] Connected to a master server at 127.3.12.188:42903
08:21:20.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.159065  8065 heartbeater.cc:461] Registering TS with master...
08:21:20.160 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f80bc8516214471ca7ea7896026da4da(127.3.12.129:46667)] cleaning up while in state READY due to: connection closed
08:21:20.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.160641  8065 heartbeater.cc:507] Master 127.3.12.188:42903 requested a full tablet report, sending...
08:21:20.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.159682  7866 raft_consensus.cc:359] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:20.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.166069  7866 raft_consensus.cc:740] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5f3bbdb1704cba8076e499c9ed1127, State: Initialized, Role: FOLLOWER
08:21:20.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.166733  8071 heartbeater.cc:344] Connected to a master server at 127.3.12.190:33597
08:21:20.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.167150  8071 heartbeater.cc:461] Registering TS with master...
08:21:20.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.167303  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 7097
08:21:20.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.168164  8071 heartbeater.cc:507] Master 127.3.12.190:33597 requested a full tablet report, sending...
08:21:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.172744  7866 consensus_queue.cc:260] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09d29cb0944901a7b67274723e6a6b" member_type: VOTER last_known_addr { host: "127.3.12.131" port: 42197 } } peers { permanent_uuid: "f80bc8516214471ca7ea7896026da4da" member_type: VOTER last_known_addr { host: "127.3.12.129" port: 46667 } } peers { permanent_uuid: "ea5f3bbdb1704cba8076e499c9ed1127" member_type: VOTER last_known_addr { host: "127.3.12.130" port: 37191 } }
08:21:20.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.183353  3477 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:20.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.185112  7866 ts_tablet_manager.cc:1434] T 203fe5a2ea01428e964d956caebbbf3c P ea5f3bbdb1704cba8076e499c9ed1127: Time spent starting tablet: real 0.058s	user 0.021s	sys 0.010s
08:21:20.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.185719  8069 heartbeater.cc:344] Connected to a master server at 127.3.12.189:34545
08:21:20.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.186084  8069 heartbeater.cc:461] Registering TS with master...
08:21:20.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.187114  8069 heartbeater.cc:507] Master 127.3.12.189:34545 requested a full tablet report, sending...
08:21:20.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.187718  3477 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.12.130:60717
08:21:20.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.189505  3203 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:20.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.193202  8065 heartbeater.cc:499] Master 127.3.12.188:42903 was elected leader, sending a full tablet report...
08:21:20.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.194135  3339 ts_manager.cc:194] Re-registered known tserver with Master: ea5f3bbdb1704cba8076e499c9ed1127 (127.3.12.130:37191)
08:21:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.221915  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 7822
08:21:20.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.266981  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 7486
08:21:20.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.311434  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 3149
08:21:20.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.358419  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 3266
08:21:20.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:21:20.403046  3122 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskjuwuwQ/build/tsan/bin/kudu with pid 3406
08:21:21.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:21:21Z chronyd exiting

Time: 72.613

OK (1 test)

