JUnit version 4.13.2
.08:00:46.308 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783497646289
08:00:46.375 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
08:00:46.445 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
08:00:46.459 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task2tL2Mb/test-tmp
08:00:46.466 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
08:00:46.470 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task2tL2Mb/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
08:00:46.657 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

08:00:46.852 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:00:46.854 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

08:00:46.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:46.860572  3942 mini_oidc.cc:150] Starting JWKS server
08:00:46.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:46.864319  3942 webserver.cc:533] Webserver started at http://127.0.0.1:41251/ using document root <none> and password file <none>
08:00:46.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:46.864833  3942 mini_oidc.cc:186] Starting OIDC Discovery server
08:00:46.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:46.867077  3942 webserver.cc:533] Webserver started at http://127.0.0.1:38353/ using document root <none> and password file <none>
08:00:46.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T08:00:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
08:00:46.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T08:00:46Z Disabled control of system clock
08:00:46.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:46.930387  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:46.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:46.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/wal
08:00:46.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data
08:00:46.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:46.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:46.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:46.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:46.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/logs
08:00:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data/info.pb
08:00:46.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:46.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:46.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:46.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:00:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:46.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:00:46.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:00:46.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.190:44599
08:00:46.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.190
08:00:46.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:46.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:46.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:00:46.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:46.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:00:46.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41251/jwks.json with env {}
08:00:47.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.269204  3954 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:47.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.269793  3954 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:47.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.270236  3954 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:47.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.303470  3954 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:00:47.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.303864  3954 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:47.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.304193  3954 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:00:47.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.304533  3954 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:00:47.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.304903  3954 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:00:47.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.305203  3954 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41251/jwks.json
08:00:47.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:47.343497  3954 master_runner.cc:387] Master server non-default flags:
08:00:47.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:47.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:47.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:47.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:47.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data
08:00:47.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/wal
08:00:47.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:00:47.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:47.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:47.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:47.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:00:47.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.190:44599
08:00:47.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:00:47.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:47.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:00:47.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41251/jwks.json
08:00:47.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:47.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:47.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data/info.pb
08:00:47.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.190
08:00:47.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:47.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:47.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:47.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:47.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:47.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:47.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/logs
08:00:47.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:47.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:47.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:47.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:00:47.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:47.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:00:47.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:47.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:00:47.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:00:47.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:00:47.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:47.345290  3954 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:47.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:47.347679  3954 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:00:47.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.362783  3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:48.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:47.362792  3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:48.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.578740  3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:48.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.581121  3962 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1217 milliseconds
08:00:48.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.581830  3954 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.220s	user 0.385s	sys 0.826s
08:00:48.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.582216  3954 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.220s	user 0.385s	sys 0.826s
08:00:48.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.582515  3954 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:00:48.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.584132  3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:48.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.587407  3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:48.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.589291  3954 hybrid_clock.cc:648] HybridClock initialized: now 1783497648589246 us; error 41 us; skew 500 ppm
08:00:48.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.590515  3954 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:48.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.599709  3954 webserver.cc:533] Webserver started at http://127.3.217.190:33787/ using document root <none> and password file <none>
08:00:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.601233  3954 fs_manager.cc:362] Metadata directory not provided
08:00:48.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.601574  3954 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:48.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.602222  3954 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:00:48.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.609443  3954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data/instance:
08:00:48.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3ea4801bae0f4eb68f23e9f0fb092ace"
08:00:48.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:48 on dist-test-slave-d17k"
08:00:48.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.611115  3954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/wal/instance:
08:00:48.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3ea4801bae0f4eb68f23e9f0fb092ace"
08:00:48.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:48 on dist-test-slave-d17k"
08:00:48.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.621492  3954 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.007s	sys 0.004s
08:00:48.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.629690  3970 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:48.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.631146  3954 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.004s	sys 0.000s
08:00:48.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.631567  3954 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/wal
08:00:48.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3ea4801bae0f4eb68f23e9f0fb092ace"
08:00:48.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:48 on dist-test-slave-d17k"
08:00:48.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.632053  3954 fs_report.cc:389] FS layout report
08:00:48.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:48.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/wal
08:00:48.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/wal
08:00:48.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data/data
08:00:48.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:48.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:48.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:48.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:48.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:48.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:48.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:48.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:48.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:48.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:48.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:48.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.713375  3954 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:48.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.715533  3954 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:48.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.716138  3954 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:48.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.788803  3954 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.190:44599
08:00:48.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.788884  4021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.190:44599 every 8 connection(s)
08:00:48.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.791847  3954 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/data/info.pb
08:00:48.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.798060  4022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:00:48.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.802309  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 3954
08:00:48.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.802830  3942 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-0/wal/instance
08:00:48.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.808023  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:48.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:48.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/wal
08:00:48.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data
08:00:48.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:48.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:48.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:48.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:48.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:48.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:48.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/logs
08:00:48.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data/info.pb
08:00:48.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:48.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:48.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:48.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:48.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:48.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:48.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:48.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:48.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:00:48.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:48.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:00:48.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:00:48.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.189:36633
08:00:48.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.189
08:00:48.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:48.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:48.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:48.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:48.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:48.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:00:48.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:48.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:00:48.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41251/jwks.json with env {}
08:00:48.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.812584  4022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } has no permanent_uuid. Determining permanent_uuid...
08:00:48.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.840351  4022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
08:00:48.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.843425  3971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:48.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.847971  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.189:36633: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:48.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.897557  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } attempt: 1
08:00:48.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.902249  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.189:36633: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:48.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:48.970708  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } attempt: 2
08:00:48.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:48.975363  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.189:36633: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:49.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:49.066888  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } attempt: 3
08:00:49.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.071699  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.189:36633: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:49.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.138355  4024 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:49.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.138913  4024 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:49.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.139366  4024 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:49.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.172024  4024 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:00:49.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.172318  4024 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:49.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.172593  4024 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:00:49.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.172793  4024 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:00:49.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.172972  4024 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:00:49.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.173250  4024 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41251/jwks.json
08:00:49.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:49.215181  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } attempt: 4
08:00:49.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:49.217155  4024 master_runner.cc:387] Master server non-default flags:
08:00:49.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:49.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:49.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:49.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:49.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data
08:00:49.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/wal
08:00:49.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:00:49.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:49.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:49.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:49.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:00:49.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.189:36633
08:00:49.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:00:49.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:49.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:00:49.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41251/jwks.json
08:00:49.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:49.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:49.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data/info.pb
08:00:49.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.189
08:00:49.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:49.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:49.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:49.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:49.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:49.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:49.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/logs
08:00:49.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:49.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:49.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:49.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:00:49.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:49.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:00:49.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:49.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:00:49.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:00:49.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:00:49.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:49.218895  4024 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:49.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.220031  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.189:36633: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:49.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:49.221261  4024 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:00:49.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.235762  4033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:49.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:49.519716  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } attempt: 5
08:00:49.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.529651  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.189:36633: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:50.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.077268  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } attempt: 6
08:00:50.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.100234  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.189:36633: Network error: Client connection negotiation failed: client connection to 127.3.217.189:36633: connect: Connection refused (error 111)
08:00:50.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:49.236511  4034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:50.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.623881  4036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.627647  4024 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.392s	user 0.538s	sys 0.851s
08:00:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.627811  4035 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1385 milliseconds
08:00:50.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.628041  4024 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.393s	user 0.538s	sys 0.851s
08:00:50.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.628348  4024 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:00:50.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.629786  4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.635097  4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:50.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.637012  4024 hybrid_clock.cc:648] HybridClock initialized: now 1783497650636934 us; error 84 us; skew 500 ppm
08:00:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.638108  4024 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:50.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.646358  4024 webserver.cc:533] Webserver started at http://127.3.217.189:42925/ using document root <none> and password file <none>
08:00:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.647768  4024 fs_manager.cc:362] Metadata directory not provided
08:00:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.648105  4024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.648712  4024 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:00:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.655588  4024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data/instance:
08:00:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31df0f27364d45eb8d580fbe289396d5"
08:00:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:50 on dist-test-slave-d17k"
08:00:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.657214  4024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/wal/instance:
08:00:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31df0f27364d45eb8d580fbe289396d5"
08:00:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:50 on dist-test-slave-d17k"
08:00:50.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.668381  4024 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.008s	sys 0.000s
08:00:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.676541  4045 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.677752  4024 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
08:00:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.678038  4024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/wal
08:00:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "31df0f27364d45eb8d580fbe289396d5"
08:00:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:50 on dist-test-slave-d17k"
08:00:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.678336  4024 fs_report.cc:389] FS layout report
08:00:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/wal
08:00:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/wal
08:00:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data/data
08:00:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:50.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:50.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:50.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.754096  4024 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.755923  4024 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.756497  4024 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:50.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.841114  4024 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.189:36633
08:00:50.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.841194  4096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.189:36633 every 8 connection(s)
08:00:50.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.844172  4024 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/data/info.pb
08:00:50.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.849530  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 4024
08:00:50.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.849865  3942 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-1/wal/instance
08:00:50.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.851061  4097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:00:50.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.851774  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:50.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:50.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/wal
08:00:50.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data
08:00:50.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:50.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:50.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:50.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:50.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:50.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:50.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/logs
08:00:50.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data/info.pb
08:00:50.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:50.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:50.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:50.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:50.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:00:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:00:50.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:00:50.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.188:33435
08:00:50.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.188
08:00:50.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:50.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:50.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:50.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:50.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:50.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:00:50.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:50.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:00:50.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41251/jwks.json with env {}
08:00:50.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.866169  4097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } has no permanent_uuid. Determining permanent_uuid...
08:00:50.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.891494  4097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
08:00:50.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.908449  4097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } has no permanent_uuid. Determining permanent_uuid...
08:00:50.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.911160  4049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:50.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.915483  4097 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:50.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:50.965085  4097 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 1
08:00:50.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:50.970100  4097 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.038465  4097 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 2
08:00:51.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.043195  4097 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.134572  4097 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 3
08:00:51.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.139297  4097 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.160787  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } attempt: 7
08:00:51.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.172602  4022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } has no permanent_uuid. Determining permanent_uuid...
08:00:51.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.178519  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.187497  4099 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:51.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.188030  4099 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:51.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.188470  4099 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.221405  4099 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:00:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.221689  4099 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.221909  4099 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:00:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.222193  4099 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:00:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.222396  4099 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:00:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.222534  4099 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41251/jwks.json
08:00:51.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.236901  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 1
08:00:51.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.241128  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.260102  4099 master_runner.cc:387] Master server non-default flags:
08:00:51.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:51.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/wal
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.188:33435
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41251/jwks.json
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:51.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data/info.pb
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.188
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/logs
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:51.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.261605  4099 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:51.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.263756  4099 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:00:51.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.276540  4111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:51.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.282713  4097 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 4
08:00:51.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.290234  4097 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.322570  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 2
08:00:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.330871  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.416517  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 3
08:00:51.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.423483  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.564077  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 4
08:00:51.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.572237  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.589766  4097 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 5
08:00:51.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.597142  4097 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:51.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:51.855893  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 5
08:00:51.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.866591  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:52.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.144732  4097 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 6
08:00:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:52.156028  4097 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:52.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.419204  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 6
08:00:52.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:52.428798  4022 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.217.188:33435: Network error: Client connection negotiation failed: client connection to 127.3.217.188:33435: connect: Connection refused (error 111)
08:00:52.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.278829  4112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:52.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:51.281955  4114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:52.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:52.519586  4113 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1237 milliseconds
08:00:52.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.519699  4099 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:00:52.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.521116  4099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:52.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.524642  4099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:52.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.526067  4099 hybrid_clock.cc:648] HybridClock initialized: now 1783497652526021 us; error 63 us; skew 500 ppm
08:00:52.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.527097  4099 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:52.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.533690  4099 webserver.cc:533] Webserver started at http://127.3.217.188:40291/ using document root <none> and password file <none>
08:00:52.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.534737  4099 fs_manager.cc:362] Metadata directory not provided
08:00:52.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.534991  4099 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:52.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.535569  4099 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:00:52.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.540234  4099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data/instance:
08:00:52.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7347676f34904e7dbb43ef0f86c51ba5"
08:00:52.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:52 on dist-test-slave-d17k"
08:00:52.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.541603  4099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/wal/instance:
08:00:52.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7347676f34904e7dbb43ef0f86c51ba5"
08:00:52.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:52 on dist-test-slave-d17k"
08:00:52.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.549135  4099 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
08:00:52.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.555047  4125 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:52.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.556134  4099 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
08:00:52.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.556519  4099 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/wal
08:00:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7347676f34904e7dbb43ef0f86c51ba5"
08:00:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:52 on dist-test-slave-d17k"
08:00:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.556928  4099 fs_report.cc:389] FS layout report
08:00:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/wal
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/wal
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data/data
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:52.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:52.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:52.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:52.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:52.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:52.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.614217  4099 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:52.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.615865  4099 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:52.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.616406  4099 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:52.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.697937  4099 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.188:33435
08:00:52.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.698050  4176 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.188:33435 every 8 connection(s)
08:00:52.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.700901  4099 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/data/info.pb
08:00:52.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.701761  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 4099
08:00:52.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.702147  3942 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/master-2/wal/instance
08:00:52.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.709619  4177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:00:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.731360  4177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } has no permanent_uuid. Determining permanent_uuid...
08:00:52.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.757822  4177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
08:00:52.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.768292  4177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } has no permanent_uuid. Determining permanent_uuid...
08:00:52.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.786980  4177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5: Bootstrap starting.
08:00:52.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.794752  4177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5: Neither blocks nor log segments found. Creating new log.
08:00:52.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.796741  4177 log.cc:826] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5: Log is configured to *not* fsync() on all Append() calls
08:00:52.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.802215  4177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5: No bootstrap required, opened a new log
08:00:52.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.823485  4177 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:52.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.824174  4177 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:00:52.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.824434  4177 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7347676f34904e7dbb43ef0f86c51ba5, State: Initialized, Role: FOLLOWER
08:00:52.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.825426  4177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:52.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.828532  4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:52.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.829762  4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: This master's current role is: FOLLOWER
08:00:52.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.830468  4177 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: configured and running, proceeding with master startup.
08:00:52.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:52.868566  4197 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:00:52.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:52.869534  4197 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:00:52.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:52.872002  4131 tablet.cc:2406] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:00:52.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.855752  4099 master_runner.cc:187] Error getting master registration for 127.3.217.190:44599: OK, instance_id { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" instance_seqno: 1783497648769791 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:00:52.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:52.886094  4099 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
08:00:53.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.173710  4186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:00:53.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.174374  4186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:53.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.177033  4186 leader_election.cc:290] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ea4801bae0f4eb68f23e9f0fb092ace (127.3.217.190:44599), 31df0f27364d45eb8d580fbe289396d5 (127.3.217.189:36633)
08:00:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.216593  4097 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 7
08:00:53.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.240989  4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: Bootstrap starting.
08:00:53.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.249197  4097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: Neither blocks nor log segments found. Creating new log.
08:00:53.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.251579  4097 log.cc:826] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: Log is configured to *not* fsync() on all Append() calls
08:00:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.257620  4097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: No bootstrap required, opened a new log
08:00:53.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.291858  4097 raft_consensus.cc:359] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:53.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.293206  4097 raft_consensus.cc:385] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:00:53.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.293506  4097 raft_consensus.cc:740] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31df0f27364d45eb8d580fbe289396d5, State: Initialized, Role: FOLLOWER
08:00:53.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.294449  4097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.297479  4201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.299376  4201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [sys.catalog]: This master's current role is: FOLLOWER
08:00:53.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.300258  4097 sys_catalog.cc:565] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [sys.catalog]: configured and running, proceeding with master startup.
08:00:53.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.300680  4072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7347676f34904e7dbb43ef0f86c51ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31df0f27364d45eb8d580fbe289396d5" is_pre_election: true
08:00:53.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.301605  4072 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7347676f34904e7dbb43ef0f86c51ba5 in term 0.
08:00:53.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.303311  4126 leader_election.cc:304] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31df0f27364d45eb8d580fbe289396d5, 7347676f34904e7dbb43ef0f86c51ba5; no voters: 
08:00:53.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.304466  4186 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
08:00:53.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.304963  4186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:00:53.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.305326  4186 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 0 FOLLOWER]: Advancing to term 1
08:00:53.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.328436  4186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:53.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.334717  4186 leader_election.cc:290] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [CANDIDATE]: Term 1 election: Requested vote from peers 3ea4801bae0f4eb68f23e9f0fb092ace (127.3.217.190:44599), 31df0f27364d45eb8d580fbe289396d5 (127.3.217.189:36633)
08:00:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.336433  4072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7347676f34904e7dbb43ef0f86c51ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31df0f27364d45eb8d580fbe289396d5"
08:00:53.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.337008  4072 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [term 0 FOLLOWER]: Advancing to term 1
08:00:53.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.345635  4072 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7347676f34904e7dbb43ef0f86c51ba5 in term 1.
08:00:53.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.347522  4126 leader_election.cc:304] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31df0f27364d45eb8d580fbe289396d5, 7347676f34904e7dbb43ef0f86c51ba5; no voters: 
08:00:53.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.348942  4186 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 1 FOLLOWER]: Leader election won for term 1
08:00:53.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.374215  4186 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [term 1 LEADER]: Becoming Leader. State: Replica: 7347676f34904e7dbb43ef0f86c51ba5, State: Running, Role: LEADER
08:00:53.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.375864  4186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:53.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:53.377496  4212 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:00:53.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:53.377959  4212 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:00:53.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.333995  4024 master_runner.cc:187] Error getting master registration for 127.3.217.190:44599: OK, instance_id { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" instance_seqno: 1783497648769791 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:00:53.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.386400  4024 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
08:00:53.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.393316  4213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7347676f34904e7dbb43ef0f86c51ba5. Latest consensus state: current_term: 1 leader_uuid: "7347676f34904e7dbb43ef0f86c51ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.395182  4213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: This master's current role is: LEADER
08:00:53.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.399008  4216 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
08:00:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.407147  4216 catalog_manager.cc:1520] Initializing Kudu cluster ID...
08:00:53.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.428122  4072 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [term 1 FOLLOWER]: Refusing update from remote peer 7347676f34904e7dbb43ef0f86c51ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:00:53.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.431739  4213 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:00:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.459192  4201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7347676f34904e7dbb43ef0f86c51ba5. Latest consensus state: current_term: 1 leader_uuid: "7347676f34904e7dbb43ef0f86c51ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.459970  4201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [sys.catalog]: This master's current role is: FOLLOWER
08:00:53.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.462229  4022 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } attempt: 7
08:00:53.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.465345  4217 mvcc.cc:204] Tried to move back new op lower bound from 7305206388413976576 to 7305206388240019456. Current Snapshot: MvccSnapshot[applied={T|T < 7305206388413976576}]
08:00:53.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.476202  4213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7347676f34904e7dbb43ef0f86c51ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.476905  4213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: This master's current role is: LEADER
08:00:53.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.478119  4218 mvcc.cc:204] Tried to move back new op lower bound from 7305206388413976576 to 7305206388240019456. Current Snapshot: MvccSnapshot[applied={T|T < 7305206388413976576}]
08:00:53.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.482328  4216 catalog_manager.cc:1383] Generated new cluster ID: d5fa83964bbd4124ba7f93707f74473f
08:00:53.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.482713  4216 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
08:00:53.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.486352  4201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7347676f34904e7dbb43ef0f86c51ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.487087  4201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5 [sys.catalog]: This master's current role is: FOLLOWER
08:00:53.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.513145  4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace: Bootstrap starting.
08:00:53.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.521394  4022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace: Neither blocks nor log segments found. Creating new log.
08:00:53.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.523684  4022 log.cc:826] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace: Log is configured to *not* fsync() on all Append() calls
08:00:53.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.535538  4216 catalog_manager.cc:1406] Generated new certificate authority record
08:00:53.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.538040  4216 catalog_manager.cc:1540] Loading token signing keys...
08:00:53.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.552263  4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace: No bootstrap required, opened a new log
08:00:53.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.585456  4216 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5: Generated new TSK 0
08:00:53.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.586623  4216 catalog_manager.cc:1550] Initializing in-progress tserver states...
08:00:53.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.594901  4022 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:53.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.596136  4022 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:00:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.596673  4022 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea4801bae0f4eb68f23e9f0fb092ace, State: Initialized, Role: FOLLOWER
08:00:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.598022  4022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } }
08:00:53.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.606490  4224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.610824  4224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [sys.catalog]: This master's current role is: FOLLOWER
08:00:53.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.611280  3997 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7347676f34904e7dbb43ef0f86c51ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" is_pre_election: true
08:00:53.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.612090  3995 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [term 0 FOLLOWER]: Advancing to term 1
08:00:53.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.610630  3996 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7347676f34904e7dbb43ef0f86c51ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace"
08:00:53.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.619403  3995 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [term 1 FOLLOWER]: Refusing update from remote peer 7347676f34904e7dbb43ef0f86c51ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:00:53.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.610466  4022 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [sys.catalog]: configured and running, proceeding with master startup.
08:00:53.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.620993  4186 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
08:00:53.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.646888  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:53.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:53.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:00:53.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:00:53.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:53.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:0
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:53.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:53.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.732373  4224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7347676f34904e7dbb43ef0f86c51ba5. Latest consensus state: current_term: 1 leader_uuid: "7347676f34904e7dbb43ef0f86c51ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.733289  4224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [sys.catalog]: This master's current role is: FOLLOWER
08:00:53.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.730785  4233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7347676f34904e7dbb43ef0f86c51ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:53.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.735015  4233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace [sys.catalog]: This master's current role is: FOLLOWER
08:00:53.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:53.762887  4241 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
08:00:54.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.168195  4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7347676f34904e7dbb43ef0f86c51ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ea4801bae0f4eb68f23e9f0fb092ace" member_type: VOTER last_known_addr { host: "127.3.217.190" port: 44599 } } peers { permanent_uuid: "31df0f27364d45eb8d580fbe289396d5" member_type: VOTER last_known_addr { host: "127.3.217.189" port: 36633 } } peers { permanent_uuid: "7347676f34904e7dbb43ef0f86c51ba5" member_type: VOTER last_known_addr { host: "127.3.217.188" port: 33435 } } }
08:00:54.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.170401  4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7347676f34904e7dbb43ef0f86c51ba5 [sys.catalog]: This master's current role is: LEADER
08:00:54.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:54.377959  4231 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:54.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:54.379165  4231 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:54.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:54.380209  4231 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:54.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.384166  4212 catalog_manager.cc:1295] Loaded cluster ID: d5fa83964bbd4124ba7f93707f74473f
08:00:54.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.384753  4212 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: loading cluster ID for follower catalog manager: success
08:00:54.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.393174  4212 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: acquiring CA information for follower catalog manager: success
08:00:54.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.399753  4212 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 31df0f27364d45eb8d580fbe289396d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:00:54.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:54.422863  4231 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:54.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:54.423882  4231 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.129
08:00:54.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.461375  4231 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:0
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:54.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:00:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:00:54.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:00:54.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:00:54.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.463176  4231 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:54.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.465673  4231 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:00:54.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:54.482895  4251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:54.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.793138  4241 catalog_manager.cc:1295] Loaded cluster ID: d5fa83964bbd4124ba7f93707f74473f
08:00:54.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.793851  4241 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace: loading cluster ID for follower catalog manager: success
08:00:54.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.802996  4241 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace: acquiring CA information for follower catalog manager: success
08:00:54.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:54.814829  4241 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3ea4801bae0f4eb68f23e9f0fb092ace: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:00:57.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:55.885356  4250 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 4231
08:00:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:57.658946  4250 kernel_stack_watchdog.cc:198] Thread 4231 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
08:00:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:00:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:00:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:00:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:00:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:57.659905  4231 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 3.174s	user 0.592s	sys 0.891s
08:00:57.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:57.660277  4231 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 3.175s	user 0.592s	sys 0.891s
08:00:57.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:57.660658  4253 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 3174 milliseconds
08:00:57.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:54.484890  4252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:57.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.662454  4231 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:00:57.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:57.662490  4255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:57.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.666108  4231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:57.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.668709  4231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:57.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.670152  4231 hybrid_clock.cc:648] HybridClock initialized: now 1783497657670103 us; error 56 us; skew 500 ppm
08:00:57.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.670940  4231 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:57.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.677732  4231 webserver.cc:533] Webserver started at http://127.3.217.129:45893/ using document root <none> and password file <none>
08:00:57.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.678587  4231 fs_manager.cc:362] Metadata directory not provided
08:00:57.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.678772  4231 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:57.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.679152  4231 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:00:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.683478  4231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/instance:
08:00:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:00:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:57 on dist-test-slave-d17k"
08:00:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.684692  4231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal/instance:
08:00:57.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:00:57.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:57 on dist-test-slave-d17k"
08:00:57.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.692436  4231 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
08:00:57.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.698721  4264 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:57.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.700019  4231 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
08:00:57.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.700394  4231 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:00:57.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:57 on dist-test-slave-d17k"
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.700807  4231 fs_report.cc:389] FS layout report
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/data
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:57.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:57.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.777477  4231 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.779846  4231 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:57.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.780838  4231 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:57.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.783849  4231 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:57.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.788973  4231 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:00:57.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.789278  4231 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
08:00:57.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.789582  4231 ts_tablet_manager.cc:616] Registered 0 tablets
08:00:57.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.789788  4231 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:00:57.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.982064  4231 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.129:36183
08:00:57.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.982126  4376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.129:36183 every 8 connection(s)
08:00:57.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.985918  4231 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:00:57.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.990020  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 4231
08:00:57.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:57.990724  3942 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal/instance
08:00:58.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.021384  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:00:58.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:0
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:58.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:58.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:58.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:58.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:58.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.048455  4378 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:00:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.048959  4377 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:00:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.049073  4378 heartbeater.cc:461] Registering TS with master...
08:00:58.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.049450  4377 heartbeater.cc:461] Registering TS with master...
08:00:58.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.055362  4377 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:00:58.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.059468  4062 ts_manager.cc:194] Registered new tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:00:58.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.062590  4378 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:00:58.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.063156  4381 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:00:58.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.063961  4381 heartbeater.cc:461] Registering TS with master...
08:00:58.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.065122  4381 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:00:58.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.069036  3987 ts_manager.cc:194] Registered new tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:00:58.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.069665  4142 ts_manager.cc:194] Registered new tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:00:58.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.071676  4142 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.129:47463
08:00:58.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:58.399291  4387 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:58.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:58.399981  4387 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:58.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:58.400414  4387 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:58.435693  4387 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:58.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:58.436480  4387 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.130
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.473774  4387 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:58.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:58.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:0
08:00:58.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:00:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:00:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:00:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:00:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:58.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:00:58.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:00:58.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:00:58.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.475078  4387 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:58.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:58.477173  4387 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:00:58.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:58.493141  4393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:59.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:00:59.075827  4378 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:00.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:59.893494  4392 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 4387
08:01:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.127820  4387 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.635s	user 0.549s	sys 0.946s
08:01:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.128304  4387 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.635s	user 0.549s	sys 0.946s
08:01:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:00:58.495019  4394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.129686  4395 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1635 milliseconds
08:01:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.131211  4387 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.131295  4396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:00.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.135071  4387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:00.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.137338  4387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:00.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.138713  4387 hybrid_clock.cc:648] HybridClock initialized: now 1783497660138677 us; error 44 us; skew 500 ppm
08:01:00.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.139575  4387 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:00.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.145596  4387 webserver.cc:533] Webserver started at http://127.3.217.130:43807/ using document root <none> and password file <none>
08:01:00.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.146617  4387 fs_manager.cc:362] Metadata directory not provided
08:01:00.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.146878  4387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:00.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.147454  4387 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:01:00.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.151968  4387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/instance:
08:01:00.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:00.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:00 on dist-test-slave-d17k"
08:01:00.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.152995  4387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal/instance:
08:01:00.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:00.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:00 on dist-test-slave-d17k"
08:01:00.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.160231  4387 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.001s
08:01:00.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.165783  4404 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.166922  4387 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
08:01:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.167245  4387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:00 on dist-test-slave-d17k"
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.167548  4387 fs_report.cc:389] FS layout report
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/data
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:00.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:00.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.221706  4387 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:00.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.223274  4387 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:00.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.223670  4387 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:00.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.226235  4387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:00.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.230417  4387 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:01:00.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.230654  4387 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:01:00.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.230842  4387 ts_tablet_manager.cc:616] Registered 0 tablets
08:01:00.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.230974  4387 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:01:00.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.381093  4387 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.130:44033
08:01:00.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.381173  4516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.130:44033 every 8 connection(s)
08:01:00.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.383738  4387 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:00.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.396162  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 4387
08:01:00.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.396732  3942 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal/instance
08:01:00.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.419147  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:00.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:00.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:00.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:0
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:00.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:00.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.442203  4518 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:00.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.442782  4518 heartbeater.cc:461] Registering TS with master...
08:01:00.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.444334  4518 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:00.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.444950  4517 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:00.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.445288  4517 heartbeater.cc:461] Registering TS with master...
08:01:00.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.446205  4517 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:00.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.447644  4142 ts_manager.cc:194] Registered new tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:00.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.448815  4519 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:00.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.449512  4519 heartbeater.cc:461] Registering TS with master...
08:01:00.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.449805  4142 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.130:47343
08:01:00.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.449801  4062 ts_manager.cc:194] Registered new tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:00.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.450505  4519 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:00.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.453119  3987 ts_manager.cc:194] Registered new tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:00.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.762892  4527 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:00.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.763479  4527 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:00.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.764199  4527 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:00.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.799419  4527 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:00.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.800501  4527 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.131
08:01:00.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.839470  4527 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:0
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:00.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:00.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.841246  4527 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:00.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:00.843076  4527 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:00.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.856717  4533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:01.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:01.454061  4518 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:02.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:02.261200  4532 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 4527
08:01:02.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:02.556552  4532 kernel_stack_watchdog.cc:198] Thread 4527 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
08:01:02.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:01:02.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:01:02.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:01:02.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:01:02.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:00.859311  4534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:02.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:02.557719  4527 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.700s	user 0.597s	sys 1.054s
08:01:02.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:02.558071  4527 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.701s	user 0.597s	sys 1.054s
08:01:02.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:02.561311  4535 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1698 milliseconds
08:01:02.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:02.563021  4536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:02.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.563078  4527 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:02.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.564309  4527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:02.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.566352  4527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:02.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.567689  4527 hybrid_clock.cc:648] HybridClock initialized: now 1783497662567651 us; error 44 us; skew 500 ppm
08:01:02.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.568466  4527 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:02.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.574687  4527 webserver.cc:533] Webserver started at http://127.3.217.131:39685/ using document root <none> and password file <none>
08:01:02.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.575522  4527 fs_manager.cc:362] Metadata directory not provided
08:01:02.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.575719  4527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:02.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.576189  4527 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:01:02.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.580479  4527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/instance:
08:01:02.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:02.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:02 on dist-test-slave-d17k"
08:01:02.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.581601  4527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal/instance:
08:01:02.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:02.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:02 on dist-test-slave-d17k"
08:01:02.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.588800  4527 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.001s
08:01:02.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.594561  4544 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:02.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.595924  4527 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.003s
08:01:02.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.596315  4527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:02.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:02 on dist-test-slave-d17k"
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.596693  4527 fs_report.cc:389] FS layout report
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/data
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:02.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:02.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:02.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:02.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:02.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:02.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.651517  4527 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:02.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.653213  4527 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:02.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.653790  4527 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:02.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.656435  4527 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:02.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.661341  4527 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:01:02.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.661577  4527 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:01:02.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.661911  4527 ts_tablet_manager.cc:616] Registered 0 tablets
08:01:02.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.662114  4527 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:01:02.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.827611  4527 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.131:33863
08:01:02.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.827710  4656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.131:33863 every 8 connection(s)
08:01:02.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.830513  4527 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:02.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.845793  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 4527
08:01:02.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.846334  3942 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal/instance
08:01:02.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.889549  4657 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:02.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.890017  4658 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:02.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.890158  4657 heartbeater.cc:461] Registering TS with master...
08:01:02.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.890380  4658 heartbeater.cc:461] Registering TS with master...
08:01:02.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.892011  4658 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:02.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.892122  4657 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:02.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.895180  4141 ts_manager.cc:194] Registered new tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:02.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.897136  4659 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:02.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.897498  4659 heartbeater.cc:461] Registering TS with master...
08:01:02.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.897662  4141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.131:48665
08:01:02.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.898677  4659 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:02.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.900065  4062 ts_manager.cc:194] Registered new tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:02.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.902540  3987 ts_manager.cc:194] Registered new tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:02.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:02.906186  3942 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
08:01:02.918 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:01:02.920 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

08:01:02.929 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.3.217.190"
      port: 44599
    }
    bound_http_address {
      host: "127.3.217.190"
      port: 33787
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.3.217.189"
      port: 36633
    }
    bound_http_address {
      host: "127.3.217.189"
      port: 42925
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.3.217.188"
      port: 33435
    }
    bound_http_address {
      host: "127.3.217.188"
      port: 40291
    }
  }
}

08:01:02.932 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

08:01:02.934 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.3.217.129"
      port: 36183
    }
    bound_http_address {
      host: "127.3.217.129"
      port: 45893
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.3.217.130"
      port: 44033
    }
    bound_http_address {
      host: "127.3.217.130"
      port: 43807
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.3.217.131"
      port: 33863
    }
    bound_http_address {
      host: "127.3.217.131"
      port: 39685
    }
  }
}

08:01:02.935 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
08:01:03.128 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
08:01:03.184 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
08:01:03.358 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.217.190:44599(127.3.217.190:44599)] connecting to peer
08:01:03.422 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.217.189:36633(127.3.217.189:36633)] connecting to peer
08:01:03.425 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.217.188:33435(127.3.217.188:33435)] connecting to peer
08:01:03.451 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.3.217.189:36633(127.3.217.189:36633)] Successfully connected to peer
08:01:03.452 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.3.217.190:44599(127.3.217.190:44599)] Successfully connected to peer
08:01:03.455 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.3.217.188:33435(127.3.217.188:33435)] Successfully connected to peer
08:01:03.565 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.3.217.189
08:01:03.565 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.3.217.190
08:01:03.565 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.3.217.188
08:01:03.810 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x4e1df28b, L:/127.0.0.1:45512 - R:/127.3.217.190:44599] using SASL/PLAIN
08:01:03.810 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xf68503d7, L:/127.0.0.1:39320 - R:/127.3.217.189:36633] using SASL/PLAIN
08:01:03.811 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x0dea5b6f, L:/127.0.0.1:41296 - R:/127.3.217.188:33435] using SASL/PLAIN
08:01:03.863 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
08:01:03.864 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=551), Traces: [0ms] refreshing cache from master, [26ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.217.190:44599, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.217.189:36633, [107ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.217.188:33435, [514ms] Sub RPC ConnectToMaster: received response from server master-127.3.217.189:36633: OK, [514ms] Sub RPC ConnectToMaster: received response from server master-127.3.217.188:33435: OK, [516ms] Sub RPC ConnectToMaster: received response from server master-127.3.217.190:44599: OK, deferred=null) after lookup
08:01:03.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:03.902793  4658 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:03.919548  4141 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41296:
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297"
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:01:03.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:01:03.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:01:03.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
08:01:03.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
08:01:03.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
08:01:03.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
08:01:03.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
08:01:03.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:01:03.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:01:03.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:03.924274  4141 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
08:01:04.013 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table fc32fa4ba7a44ca19e9c1e9c6bcb981a
08:01:04.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.051241  4592 tablet_service.cc:1511] Processing CreateTablet for tablet bd21da974ee444b4adcfa3d96158740f (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297 [id=fc32fa4ba7a44ca19e9c1e9c6bcb981a]), partition=RANGE (key) PARTITION UNBOUNDED
08:01:04.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.054304  4592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd21da974ee444b4adcfa3d96158740f. 1 dirs total, 0 dirs full, 0 dirs failed
08:01:04.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.109987  4312 tablet_service.cc:1511] Processing CreateTablet for tablet bd21da974ee444b4adcfa3d96158740f (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297 [id=fc32fa4ba7a44ca19e9c1e9c6bcb981a]), partition=RANGE (key) PARTITION UNBOUNDED
08:01:04.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.112802  4312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd21da974ee444b4adcfa3d96158740f. 1 dirs total, 0 dirs full, 0 dirs failed
08:01:04.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.133425  4692 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap starting.
08:01:04.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.147828  4452 tablet_service.cc:1511] Processing CreateTablet for tablet bd21da974ee444b4adcfa3d96158740f (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297 [id=fc32fa4ba7a44ca19e9c1e9c6bcb981a]), partition=RANGE (key) PARTITION UNBOUNDED
08:01:04.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.150871  4452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd21da974ee444b4adcfa3d96158740f. 1 dirs total, 0 dirs full, 0 dirs failed
08:01:04.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.174815  4692 tablet_bootstrap.cc:654] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Neither blocks nor log segments found. Creating new log.
08:01:04.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.180665  4692 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Log is configured to *not* fsync() on all Append() calls
08:01:04.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.184315  4693 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap starting.
08:01:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.199934  4692 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: No bootstrap required, opened a new log
08:01:04.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.200696  4692 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent bootstrapping tablet: real 0.069s	user 0.014s	sys 0.025s
08:01:04.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.209738  4693 tablet_bootstrap.cc:654] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Neither blocks nor log segments found. Creating new log.
08:01:04.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.217535  4693 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Log is configured to *not* fsync() on all Append() calls
08:01:04.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.230224  4696 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap starting.
08:01:04.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.234300  4693 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: No bootstrap required, opened a new log
08:01:04.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.234889  4693 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent bootstrapping tablet: real 0.052s	user 0.021s	sys 0.016s
08:01:04.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.242547  4692 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.243589  4692 raft_consensus.cc:385] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:01:04.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.243932  4692 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Initialized, Role: FOLLOWER
08:01:04.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.245265  4692 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.255074  4696 tablet_bootstrap.cc:654] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Neither blocks nor log segments found. Creating new log.
08:01:04.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.259296  4696 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Log is configured to *not* fsync() on all Append() calls
08:01:04.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.259687  4692 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent starting tablet: real 0.059s	user 0.031s	sys 0.024s
08:01:04.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.283416  4696 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: No bootstrap required, opened a new log
08:01:04.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.284152  4696 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent bootstrapping tablet: real 0.056s	user 0.013s	sys 0.020s
08:01:04.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.287119  4693 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.288420  4693 raft_consensus.cc:385] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:01:04.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.288856  4693 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4f3ebf8a7415ca003f30ddf138eea, State: Initialized, Role: FOLLOWER
08:01:04.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.290215  4693 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.296365  4693 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent starting tablet: real 0.061s	user 0.032s	sys 0.017s
08:01:04.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:04.302328  4384 tablet.cc:2406] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:01:04.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.319038  4696 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.319743  4696 raft_consensus.cc:385] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:01:04.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.320042  4696 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Initialized, Role: FOLLOWER
08:01:04.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.320927  4696 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.325925  4696 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent starting tablet: real 0.041s	user 0.033s	sys 0.009s
08:01:04.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:04.356735  4664 tablet.cc:2406] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:01:04.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:04.440488  4524 tablet.cc:2406] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:01:04.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.655259  4697 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:01:04.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.655819  4697 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.658593  4697 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.671020  4332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" is_pre_election: true
08:01:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.672163  4472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da" is_pre_election: true
08:01:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.672214  4332 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 0.
08:01:04.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.672787  4472 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 0.
08:01:04.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.673686  4545 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, 802ab38827f94c91a1d169aa96b11b7d; no voters: 
08:01:04.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.674366  4697 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Leader pre-election won for term 1
08:01:04.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.674726  4697 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:01:04.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.675019  4697 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 0 FOLLOWER]: Advancing to term 1
08:01:04.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.679385  4697 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.681103  4697 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 1 election: Requested vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:04.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.682044  4472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:04.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.682430  4472 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 0 FOLLOWER]: Advancing to term 1
08:01:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.682817  4332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.683410  4332 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 0 FOLLOWER]: Advancing to term 1
08:01:04.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.687543  4472 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 1.
08:01:04.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.688006  4332 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 1.
08:01:04.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.688568  4545 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da; no voters: 
08:01:04.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.689251  4697 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Leader election won for term 1
08:01:04.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.690831  4697 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 LEADER]: Becoming Leader. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Running, Role: LEADER
08:01:04.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.692332  4697 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:04.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:04.706197  4141 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131). New cstate: current_term: 1 leader_uuid: "802ab38827f94c91a1d169aa96b11b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: UNKNOWN } } }
08:01:05.063 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table fc32fa4ba7a44ca19e9c1e9c6bcb981a
08:01:05.083 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:05.090 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:05.091 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [10ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=null) after lookup
08:01:05.105 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:05.123 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] Successfully connected to peer
08:01:05.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:05.179109  4697 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
08:01:05.187 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x80965851, L:/127.0.0.1:59798 - R:/127.3.217.131:33863] using TOKEN/null
08:01:05.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:05.211035  4472 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 1 FOLLOWER]: Refusing update from remote peer 802ab38827f94c91a1d169aa96b11b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:01:05.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:05.218700  4697 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:01:05.371 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:05.374 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.131:33863
08:01:05.376 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:05.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:05.376722  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 4527
08:01:05.402 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:05.405 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] handling channelInactive
08:01:05.406 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state READY due to: connection closed
08:01:05.406 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: connection closed
08:01:05.419 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:05.419 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:05.420 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [3ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1101059008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@38a400e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ce327c4 -> wakeup thread Time-limited test)) after lookup
08:01:05.420 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:05.429 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.429 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.430 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.438 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:05.438 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:05.438 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [3ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [17ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [27ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [27ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1101059008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@38a400e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ce327c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640))) after lookup
08:01:05.439 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:05.441 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.441 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.442 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.479 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:05.480 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:05.481 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [3ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [17ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [27ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [27ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [39ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [39ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [71ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1101059008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@38a400e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ce327c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931))) after lookup
08:01:05.482 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:05.484 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.484 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.484 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:05.499 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:05.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=96), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [3ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [17ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [27ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [27ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [39ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [39ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [71ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [78ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [81ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [81ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [91ms] refreshing cache from master, [91ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [95ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1101059008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@38a400e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931) -> (continuation of Deferred@721450048 after retry RPC@1410494984), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ce327c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931) -> (continuation of Deferred@721450048 after retry RPC@1410494984))) after lookup
08:01:05.501 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:05.503 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.503 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.503 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.740 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:05.740 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:05.741 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=337), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [3ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [17ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [27ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [27ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [39ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [39ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [71ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [78ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [81ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [81ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [91ms] refreshing cache from master, [91ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [95ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [97ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [100ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [100ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [331ms] refreshing cache from master, [332ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [336ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1101059008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@38a400e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931) -> (continuation of Deferred@721450048 after retry RPC@1410494984) -> (continuation of Deferred@1869098249 after retry RPC@1709499010), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ce327c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931) -> (continuation of Deferred@721450048 after retry RPC@1410494984) -> (continuation of Deferred@1869098249 after retry RPC@1709499010))) after lookup
08:01:05.742 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:05.743 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.743 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:05.743 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:06.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.813447  4737 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:06.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.813719  4738 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:06.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.813906  4737 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:06.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.814132  4738 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:06.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.816762  4737 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:06.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.824338  4738 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:06.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:06.830394  4407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:06.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:06.830901  4267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:06.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.842406  4332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" is_pre_election: true
08:01:06.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:06.842953  4407 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:06.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.843467  4332 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 1.
08:01:06.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.844841  4405 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:06.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.845986  4738 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 1 FOLLOWER]: Leader pre-election won for term 2
08:01:06.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.846287  4738 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:06.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.846635  4738 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 1 FOLLOWER]: Advancing to term 2
08:01:06.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.850055  4472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da" is_pre_election: true
08:01:06.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:06.850216  4267 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:06.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.853478  4738 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:06.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.854804  4472 raft_consensus.cc:2393] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ee4f3ebf8a7415ca003f30ddf138eea in current term 2: Already voted for candidate cb31ba83425d4f48bbc683abe7c675da in this term.
08:01:06.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.856736  4332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:06.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.857255  4332 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 1 FOLLOWER]: Advancing to term 2
08:01:06.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.859275  4265 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea; no voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da
08:01:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:06.860502  4407 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:06.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.861164  4738 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 2 election: Requested vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:06.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.863974  4332 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 2.
08:01:06.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.864501  4737 raft_consensus.cc:2749] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
08:01:06.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.865103  4405 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:06.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.866386  4738 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 FOLLOWER]: Leader election won for term 2
08:01:06.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.868378  4738 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 LEADER]: Becoming Leader. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Running, Role: LEADER
08:01:06.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.869552  4738 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:06.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:06.879307  4141 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da reported cstate change: term changed from 1 to 2, leader changed from 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131) to cb31ba83425d4f48bbc683abe7c675da (127.3.217.130). New cstate: current_term: 2 leader_uuid: "cb31ba83425d4f48bbc683abe7c675da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: HEALTHY } } }
08:01:07.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:07.273466  4407 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:07.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:07.362255  4332 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Refusing update from remote peer cb31ba83425d4f48bbc683abe7c675da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
08:01:07.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:07.363629  4738 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
08:01:07.740 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:07.740 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:07.741 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2338), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [3ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [16ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [17ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [27ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [27ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [39ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [39ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [71ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [76ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [78ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [81ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [81ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [91ms] refreshing cache from master, [91ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [95ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [97ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [100ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [100ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [331ms] refreshing cache from master, [332ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [336ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [338ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [340ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [340ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [2331ms] refreshing cache from master, [2331ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [2336ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1101059008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@38a400e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931) -> (continuation of Deferred@721450048 after retry RPC@1410494984) -> (continuation of Deferred@1869098249 after retry RPC@1709499010) -> (continuation of Deferred@890203697 after retry RPC@2000513760), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3ce327c4 -> wakeup thread Time-limited test -> (continuation of Deferred@1418585599 after retry RPC@149954640) -> (continuation of Deferred@917802108 after retry RPC@2005615931) -> (continuation of Deferred@721450048 after retry RPC@1410494984) -> (continuation of Deferred@1869098249 after retry RPC@1709499010) -> (continuation of Deferred@890203697 after retry RPC@2000513760))) after lookup
08:01:07.743 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:07.746 [DEBUG - kudu-nio-1] (Connection.java:757) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] Successfully connected to peer
08:01:07.770 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xe7e12a7d, L:/127.0.0.1:35052 - R:/127.3.217.130:44033] using TOKEN/null
08:01:07.829 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.131:33863
08:01:07.830 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:07.832875  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:07.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:07.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:08.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:08.169445  4771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:08.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:08.169921  4771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:08.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:08.170369  4771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:08.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:08.203935  4771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:08.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:08.204761  4771 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.131
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:08.242882  4771 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:08.244171  4771 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:08.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:08.245782  4771 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:08.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:08.258558  4777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:09.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:08.260056  4778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:09.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:09.575803  4780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:09.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:09.578227  4779 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1314 milliseconds
08:01:09.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.578321  4771 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:09.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.579722  4771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:09.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.582186  4771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:09.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.583539  4771 hybrid_clock.cc:648] HybridClock initialized: now 1783497669583504 us; error 46 us; skew 500 ppm
08:01:09.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.584363  4771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:09.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.590750  4771 webserver.cc:533] Webserver started at http://127.3.217.131:39685/ using document root <none> and password file <none>
08:01:09.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.591661  4771 fs_manager.cc:362] Metadata directory not provided
08:01:09.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.591892  4771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:09.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.600095  4771 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
08:01:09.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.604832  4791 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:09.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.606155  4771 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:01:09.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.606467  4771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:09.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:09.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:02 on dist-test-slave-d17k"
08:01:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.608477  4771 fs_report.cc:389] FS layout report
08:01:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/data
08:01:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:09.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:09.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.674737  4771 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:09.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.676301  4771 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:09.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.676723  4771 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:09.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.679726  4771 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:09.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.685961  4798 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:09.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.695487  4771 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:09.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.695782  4771 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
08:01:09.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.696087  4771 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:09.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:09.702756  4407 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:01:09.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.703430  4771 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:09.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.703678  4771 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.007s	sys 0.001s
08:01:09.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.704223  4798 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap starting.
08:01:09.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.779165  4798 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Log is configured to *not* fsync() on all Append() calls
08:01:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.811156  4798 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:09.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.812193  4798 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap complete.
08:01:09.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.814085  4798 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent bootstrapping tablet: real 0.110s	user 0.070s	sys 0.035s
08:01:09.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.838987  4798 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:09.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.841813  4798 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Initialized, Role: FOLLOWER
08:01:09.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.843269  4798 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:09.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.849308  4798 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent starting tablet: real 0.035s	user 0.029s	sys 0.004s
08:01:09.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.896065  4771 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.131:33863
08:01:09.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.896454  4908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.131:33863 every 8 connection(s)
08:01:09.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.899196  4771 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:09.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.906064  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 4771
08:01:09.907 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:09.939 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:09.940 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.130:44033
08:01:09.940 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:01:09.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.940932  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 4387
08:01:09.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.948751  4910 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:09.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.948994  4911 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:09.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.949333  4910 heartbeater.cc:461] Registering TS with master...
08:01:09.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.949424  4911 heartbeater.cc:461] Registering TS with master...
08:01:09.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.950887  4910 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:09.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.951550  4909 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:09.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.950889  4911 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:09.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.951957  4909 heartbeater.cc:461] Registering TS with master...
08:01:09.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.952857  4909 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:09.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.955703  4142 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:09.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.956890  3987 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:09.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.957139  4062 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:09.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:09.958838  4142 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.131:59437
08:01:09.975 [DEBUG - kudu-nio-1] (Connection.java:239) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] handling channelInactive
08:01:09.975 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state READY due to: connection closed
08:01:09.975 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:09.976 [DEBUG - Time-limited test] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:09.978 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:09.978 [DEBUG - kudu-nio-2] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:09.978 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.000 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:10.000 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:10.001 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [2ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@89341365(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@61a68bec -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f0f5811 -> wakeup thread Time-limited test)) after lookup
08:01:10.002 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:10.003 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.003 [DEBUG - kudu-nio-3] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.003 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:10.019 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:10.020 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [2ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [25ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [27ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [27ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@89341365(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@61a68bec -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f0f5811 -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737))) after lookup
08:01:10.021 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:10.022 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.023 [DEBUG - kudu-nio-4] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.023 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.059 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:10.059 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:10.060 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [2ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [25ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [27ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [27ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [45ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [47ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [47ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@89341365(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@61a68bec -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f0f5811 -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264))) after lookup
08:01:10.060 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:10.061 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.062 [DEBUG - kudu-nio-5] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.062 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.178 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:10.179 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:10.179 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=203), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [2ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [25ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [27ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [27ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [45ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [47ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [47ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [84ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [86ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [86ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [198ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [202ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@89341365(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@61a68bec -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f0f5811 -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102))) after lookup
08:01:10.181 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:10.182 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.182 [DEBUG - kudu-nio-6] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.182 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.218 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:10.218 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:10.219 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=242), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [2ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [25ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [27ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [27ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [45ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [47ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [47ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [84ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [86ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [86ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [198ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [202ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [204ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [206ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [206ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [238ms] refreshing cache from master, [238ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [242ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@89341365(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@61a68bec -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102) -> (continuation of Deferred@256175161 after retry RPC@1014204711), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f0f5811 -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102) -> (continuation of Deferred@256175161 after retry RPC@1014204711))) after lookup
08:01:10.220 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:10.221 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.221 [DEBUG - kudu-nio-7] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.221 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.379 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:10.379 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:10.380 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=403), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [2ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [25ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [27ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [27ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [45ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [47ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [47ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [84ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [86ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [86ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [198ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [202ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [204ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [206ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [206ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [238ms] refreshing cache from master, [238ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [242ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [243ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [245ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [245ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [398ms] refreshing cache from master, [399ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [403ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@89341365(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@61a68bec -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102) -> (continuation of Deferred@256175161 after retry RPC@1014204711) -> (continuation of Deferred@1598826636 after retry RPC@1980930097), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f0f5811 -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102) -> (continuation of Deferred@256175161 after retry RPC@1014204711) -> (continuation of Deferred@1598826636 after retry RPC@1980930097))) after lookup
08:01:10.381 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:10.382 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.382 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.382 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:10.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:10.962083  4910 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:11.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.199549  4921 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:01:11.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.200091  4921 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:11.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.202713  4921 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:11.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:11.208071  4792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:11.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:11.212373  4792 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:11.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.213724  4332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" is_pre_election: true
08:01:11.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.215303  4792 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d; no voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da
08:01:11.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.216032  4921 raft_consensus.cc:2749] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
08:01:11.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.261552  4927 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cb31ba83425d4f48bbc683abe7c675da)
08:01:11.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.261925  4927 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:11.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.263470  4927 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:11.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:11.271003  4265 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:11.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.275717  4863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "802ab38827f94c91a1d169aa96b11b7d" is_pre_election: true
08:01:11.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.276371  4863 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee4f3ebf8a7415ca003f30ddf138eea in term 1.
08:01:11.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.277735  4267 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, 802ab38827f94c91a1d169aa96b11b7d; no voters: cb31ba83425d4f48bbc683abe7c675da
08:01:11.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.278553  4927 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Leader pre-election won for term 3
08:01:11.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.278867  4927 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Starting leader election (detected failure of leader cb31ba83425d4f48bbc683abe7c675da)
08:01:11.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.279225  4927 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 2 FOLLOWER]: Advancing to term 3
08:01:11.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.283660  4927 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:11.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.285310  4927 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 3 election: Requested vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:11.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.286244  4863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:11.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.286648  4863 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 1 FOLLOWER]: Advancing to term 3
08:01:11.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:11.288964  4265 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:11.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.293817  4863 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee4f3ebf8a7415ca003f30ddf138eea in term 3.
08:01:11.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.294775  4267 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, 802ab38827f94c91a1d169aa96b11b7d; no voters: cb31ba83425d4f48bbc683abe7c675da
08:01:11.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.295420  4927 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Leader election won for term 3
08:01:11.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.297255  4927 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 LEADER]: Becoming Leader. State: Replica: 5ee4f3ebf8a7415ca003f30ddf138eea, State: Running, Role: LEADER
08:01:11.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.298450  4927 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:11.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.308625  4142 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea reported cstate change: term changed from 2 to 3, leader changed from cb31ba83425d4f48bbc683abe7c675da (127.3.217.130) to 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129). New cstate: current_term: 3 leader_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: UNKNOWN } } }
08:01:11.720 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:11.720 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:11.721 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1744), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [2ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [25ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [27ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [27ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [45ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [47ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [47ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [83ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [84ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [86ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [86ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [198ms] refreshing cache from master, [198ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [202ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [204ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [206ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [206ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [238ms] refreshing cache from master, [238ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [242ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [243ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [245ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [245ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [398ms] refreshing cache from master, [399ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [403ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [404ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [406ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [407ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [1738ms] refreshing cache from master, [1738ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [1743ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@89341365(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@61a68bec -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102) -> (continuation of Deferred@256175161 after retry RPC@1014204711) -> (continuation of Deferred@1598826636 after retry RPC@1980930097) -> (continuation of Deferred@1018138246 after retry RPC@1587431986), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f0f5811 -> wakeup thread Time-limited test -> (continuation of Deferred@1384486160 after retry RPC@1473220737) -> (continuation of Deferred@125098426 after retry RPC@84154264) -> (continuation of Deferred@337460554 after retry RPC@822100102) -> (continuation of Deferred@256175161 after retry RPC@1014204711) -> (continuation of Deferred@1598826636 after retry RPC@1980930097) -> (continuation of Deferred@1018138246 after retry RPC@1587431986))) after lookup
08:01:11.722 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:11.723 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] Successfully connected to peer
08:01:11.753 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xc348f9ec, L:/127.0.0.1:45234 - R:/127.3.217.129:36183] using TOKEN/null
08:01:11.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.767704  4863 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 3 FOLLOWER]: Refusing update from remote peer 5ee4f3ebf8a7415ca003f30ddf138eea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
08:01:11.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:11.770025  4265 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea -> Peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Couldn't send request to peer cb31ba83425d4f48bbc683abe7c675da. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:11.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.770042  4927 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [LEADER]: Connected to new peer: Peer: permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
08:01:11.844 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.130:44033
08:01:11.845 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:11.846801  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:11.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:44033
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43807
08:01:11.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:01:11.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:11.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:11.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:11.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:12.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.207561  4955 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:12.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.208055  4955 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:12.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.208820  4955 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:12.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.214073  4265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111) [suppressed 3 similar messages]
08:01:12.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.243685  4955 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:12.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.244532  4955 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.130
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:12.281518  4955 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:44033
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43807
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:12.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:12.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:12.283185  4955 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:12.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:12.284739  4955 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:12.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.297768  4961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:13.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:12.299084  4962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:13.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:13.627562  4964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:13.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:13.629904  4963 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1326 milliseconds
08:01:13.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.630007  4955 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:13.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.631181  4955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:13.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.633582  4955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:13.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.634931  4955 hybrid_clock.cc:648] HybridClock initialized: now 1783497673634902 us; error 45 us; skew 500 ppm
08:01:13.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.635732  4955 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:13.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.642241  4955 webserver.cc:533] Webserver started at http://127.3.217.130:43807/ using document root <none> and password file <none>
08:01:13.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.643136  4955 fs_manager.cc:362] Metadata directory not provided
08:01:13.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.643342  4955 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:13.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.651309  4955 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
08:01:13.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.655892  4975 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:13.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.657191  4955 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.657509  4955 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:00 on dist-test-slave-d17k"
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.659417  4955 fs_report.cc:389] FS layout report
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/data
08:01:13.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:13.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:13.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.731204  4955 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:13.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.733014  4955 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:13.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.733515  4955 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:13.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.736354  4955 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:13.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.742645  4982 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:13.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.751327  4955 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:13.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.751593  4955 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.000s
08:01:13.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.751942  4955 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:13.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.757710  4955 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:13.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.757910  4955 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.006s	sys 0.000s
08:01:13.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.758405  4982 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap starting.
08:01:13.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.830439  4982 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Log is configured to *not* fsync() on all Append() calls
08:01:13.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.907702  4982 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:13.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.908960  4982 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap complete.
08:01:13.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.910808  4982 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent bootstrapping tablet: real 0.153s	user 0.080s	sys 0.046s
08:01:13.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.939733  4982 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:13.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.943455  4982 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Initialized, Role: FOLLOWER
08:01:13.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.944638  4982 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:13.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:13.951612  4982 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent starting tablet: real 0.040s	user 0.033s	sys 0.002s
08:01:14.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.013720  4955 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.130:44033
08:01:14.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.014034  5092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.130:44033 every 8 connection(s)
08:01:14.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.016291  4955 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:14.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.019701  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 4955
08:01:14.020 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:14.053 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:14.054 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.129:36183
08:01:14.055 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:01:14.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.055359  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 4231
08:01:14.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.077907  5093 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:14.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.078365  5093 heartbeater.cc:461] Registering TS with master...
08:01:14.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.078388  5094 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:14.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.079209  5094 heartbeater.cc:461] Registering TS with master...
08:01:14.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.079759  5093 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:14.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.080520  5094 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:14.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.083685  5095 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:14.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.084045  5095 heartbeater.cc:461] Registering TS with master...
08:01:14.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.084270  4062 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:14.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.084260  4142 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:14.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.085011  5095 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:14.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.087172  4142 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.130:52345
08:01:14.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:14.088929  3987 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:14.096 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] handling channelInactive
08:01:14.097 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state READY due to: connection closed
08:01:14.097 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:14.098 [DEBUG - Time-limited test] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:14.099 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.099 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.100 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.118 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:14.119 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:14.119 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1679576543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4628c8a5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f7a7088 -> wakeup thread Time-limited test)) after lookup
08:01:14.120 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:14.121 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.122 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.122 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.139 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:14.139 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:14.139 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [22ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [24ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [24ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1679576543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4628c8a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f7a7088 -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532))) after lookup
08:01:14.140 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:14.141 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.141 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.141 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.178 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:14.179 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:14.179 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [22ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [24ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [24ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [42ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [43ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [43ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1679576543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4628c8a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f7a7088 -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244))) after lookup
08:01:14.180 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:14.181 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.181 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.181 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.279 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:14.279 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:14.280 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=182), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [22ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [24ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [24ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [42ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [43ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [43ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [81ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [83ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [84ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [176ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [180ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1679576543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4628c8a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f7a7088 -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942))) after lookup
08:01:14.280 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:14.281 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.282 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.282 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.339 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:14.340 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:14.340 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=242), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [22ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [24ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [24ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [42ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [43ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [43ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [81ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [83ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [84ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [176ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [180ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [182ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [184ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [184ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [236ms] refreshing cache from master, [237ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [241ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1679576543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4628c8a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942) -> (continuation of Deferred@2120838007 after retry RPC@1755817847), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f7a7088 -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942) -> (continuation of Deferred@2120838007 after retry RPC@1755817847))) after lookup
08:01:14.341 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:14.342 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.342 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:14.342 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:15.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.091250  5094 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:15.398 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:15.398 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:15.399 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1301), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [22ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [24ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [24ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [42ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [43ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [43ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [81ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [83ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [84ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [176ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [180ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [182ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [184ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [184ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [236ms] refreshing cache from master, [237ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [241ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [243ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [244ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [244ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1296ms] refreshing cache from master, [1296ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [1300ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1679576543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4628c8a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942) -> (continuation of Deferred@2120838007 after retry RPC@1755817847) -> (continuation of Deferred@1437069588 after retry RPC@15624392), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f7a7088 -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942) -> (continuation of Deferred@2120838007 after retry RPC@1755817847) -> (continuation of Deferred@1437069588 after retry RPC@15624392))) after lookup
08:01:15.400 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:15.401 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:15.401 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:15.401 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:15.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.451148  5104 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4f3ebf8a7415ca003f30ddf138eea)
08:01:15.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.451538  5104 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:15.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.452977  5104 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:15.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:15.459198  4792 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:15.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.464275  5048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da" is_pre_election: true
08:01:15.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.464915  5048 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 2.
08:01:15.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.466018  4792 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da; no voters: 5ee4f3ebf8a7415ca003f30ddf138eea
08:01:15.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.466616  5104 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 3 FOLLOWER]: Leader pre-election won for term 4
08:01:15.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.466902  5104 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5ee4f3ebf8a7415ca003f30ddf138eea)
08:01:15.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.467258  5104 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 3 FOLLOWER]: Advancing to term 4
08:01:15.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.471773  5104 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:15.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.473284  5104 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 4 election: Requested vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:15.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.474743  5048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:15.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.475301  5048 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 2 FOLLOWER]: Advancing to term 4
08:01:15.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:15.476821  4792 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:15.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.481954  5048 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 4.
08:01:15.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.482971  4792 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da; no voters: 5ee4f3ebf8a7415ca003f30ddf138eea
08:01:15.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.483829  5104 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 4 FOLLOWER]: Leader election won for term 4
08:01:15.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.484805  5104 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 4 LEADER]: Becoming Leader. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Running, Role: LEADER
08:01:15.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.485622  5104 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:15.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.493690  4142 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d reported cstate change: term changed from 3 to 4, leader changed from 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129) to 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131). New cstate: current_term: 4 leader_uuid: "802ab38827f94c91a1d169aa96b11b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: UNKNOWN } } }
08:01:15.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:15.914294  4792 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d -> Peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Couldn't send request to peer 5ee4f3ebf8a7415ca003f30ddf138eea. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:15.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:15.999076  5048 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 4 FOLLOWER]: Refusing update from remote peer 802ab38827f94c91a1d169aa96b11b7d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
08:01:16.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:16.000501  5104 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
08:01:16.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:16.016748  5122 mvcc.cc:204] Tried to move back new op lower bound from 7305206463534489600 to 7305206461646381056. Current Snapshot: MvccSnapshot[applied={T|T < 7305206463534489600}]
08:01:16.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:16.345593  4792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111) [suppressed 3 similar messages]
08:01:17.259 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:17.260 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:17.260 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=3162), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [22ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [24ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [24ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [36ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [42ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [43ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [43ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [81ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [83ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [84ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [176ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [180ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [182ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [184ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [184ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [236ms] refreshing cache from master, [237ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [241ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [243ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [244ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [244ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1296ms] refreshing cache from master, [1296ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [1300ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [1301ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1303ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1303ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [3156ms] refreshing cache from master, [3157ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [3161ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1679576543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4628c8a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942) -> (continuation of Deferred@2120838007 after retry RPC@1755817847) -> (continuation of Deferred@1437069588 after retry RPC@15624392) -> (continuation of Deferred@173610 after retry RPC@1943615178), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f7a7088 -> wakeup thread Time-limited test -> (continuation of Deferred@1959158476 after retry RPC@1477751532) -> (continuation of Deferred@1067324081 after retry RPC@1113605244) -> (continuation of Deferred@1728434430 after retry RPC@654719942) -> (continuation of Deferred@2120838007 after retry RPC@1755817847) -> (continuation of Deferred@1437069588 after retry RPC@15624392) -> (continuation of Deferred@173610 after retry RPC@1943615178))) after lookup
08:01:17.266 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:17.267 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] Successfully connected to peer
08:01:17.293 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x0667fc0f, L:/127.0.0.1:57868 - R:/127.3.217.131:33863] using TOKEN/null
08:01:17.350 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.129:36183
08:01:17.350 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:17.352645  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:36183
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45893
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:17.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.666466  5139 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.666958  5139 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.667408  5139 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:17.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.701030  5139 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:17.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.701977  5139 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.129
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:17.741153  5139 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:36183
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:17.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45893
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:17.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:17.742697  5139 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:17.744717  5139 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:17.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.758205  5147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:18.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:18.226897  4792 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d -> Peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Couldn't send request to peer 5ee4f3ebf8a7415ca003f30ddf138eea. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:01:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.763505  5150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:17.759377  5148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:19.130304  5149 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1366 milliseconds
08:01:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.130424  5139 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.131995  5139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:19.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.134332  5139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:19.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.135700  5139 hybrid_clock.cc:648] HybridClock initialized: now 1783497679135650 us; error 70 us; skew 500 ppm
08:01:19.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.136512  5139 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:19.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.143400  5139 webserver.cc:533] Webserver started at http://127.3.217.129:45893/ using document root <none> and password file <none>
08:01:19.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.144440  5139 fs_manager.cc:362] Metadata directory not provided
08:01:19.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.144713  5139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:19.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.152839  5139 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
08:01:19.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.157500  5161 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:19.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.158697  5139 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
08:01:19.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.159150  5139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:57 on dist-test-slave-d17k"
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.161897  5139 fs_report.cc:389] FS layout report
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/data
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:19.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:19.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:19.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:19.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:19.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:19.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.218896  5139 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:19.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.221128  5139 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:19.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.221760  5139 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:19.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.225302  5139 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:19.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.233599  5169 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:19.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.242085  5139 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:19.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.242347  5139 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.003s	sys 0.001s
08:01:19.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.242664  5139 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:19.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.247674  5139 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:19.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.247972  5139 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
08:01:19.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.248366  5169 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap starting.
08:01:19.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.332484  5169 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Log is configured to *not* fsync() on all Append() calls
08:01:19.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.378746  5169 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:19.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.379657  5169 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap complete.
08:01:19.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.381250  5169 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent bootstrapping tablet: real 0.133s	user 0.096s	sys 0.032s
08:01:19.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.399595  5169 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:19.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.402076  5169 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4f3ebf8a7415ca003f30ddf138eea, State: Initialized, Role: FOLLOWER
08:01:19.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.403295  5169 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.407444  5169 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent starting tablet: real 0.026s	user 0.013s	sys 0.012s
08:01:19.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.432430  5139 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.129:36183
08:01:19.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.432806  5277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.129:36183 every 8 connection(s)
08:01:19.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.435175  5139 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:19.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.437837  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 5139
08:01:19.439 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:19.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.477887  5279 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:19.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.478461  5279 heartbeater.cc:461] Registering TS with master...
08:01:19.479 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:19.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.479758  5279 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:19.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.481645  5278 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:19.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.482039  5278 heartbeater.cc:461] Registering TS with master...
08:01:19.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.483158  5278 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:19.485 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.131:33863
08:01:19.485 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:19.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.486030  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 4771
08:01:19.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.487807  4141 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:19.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.490897  4141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.129:41585
08:01:19.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.492933  4061 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:19.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.501587  5280 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:19.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.502015  5280 heartbeater.cc:461] Registering TS with master...
08:01:19.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.502745  5280 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:19.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:19.506861  3987 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:19.523 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] handling channelInactive
08:01:19.523 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:19.523 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state READY due to: connection closed
08:01:19.524 [DEBUG - Time-limited test] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:19.525 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.525 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.526 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.538 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:19.538 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:19.539 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [2ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [14ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@66720331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@134e0caa -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2069e25d -> wakeup thread Time-limited test)) after lookup
08:01:19.539 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:19.541 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.541 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.542 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.558 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:19.558 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:19.559 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [2ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [14ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [18ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [19ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@66720331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@134e0caa -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2069e25d -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832))) after lookup
08:01:19.559 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:19.560 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.560 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.561 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:19.578 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:19.578 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=55), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [2ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [14ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [18ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [19ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [38ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [38ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [55ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@66720331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@134e0caa -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2069e25d -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257))) after lookup
08:01:19.579 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:19.580 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.580 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.581 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.679 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:19.679 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:19.680 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=156), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [2ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [14ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [18ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [19ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [38ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [38ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [55ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [56ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [58ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [58ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [151ms] refreshing cache from master, [151ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [155ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@66720331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@134e0caa -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2069e25d -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318))) after lookup
08:01:19.680 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:19.682 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.682 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.682 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.959 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:19.959 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:19.959 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=435), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [2ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [14ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [18ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [19ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [38ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [38ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [55ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [56ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [58ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [58ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [151ms] refreshing cache from master, [151ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [155ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [157ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [159ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [159ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [435ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@66720331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@134e0caa -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318) -> (continuation of Deferred@1877883845 after retry RPC@614449563), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2069e25d -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318) -> (continuation of Deferred@1877883845 after retry RPC@614449563))) after lookup
08:01:19.960 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:19.961 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.961 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:19.961 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:20.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.497756  5279 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:20.518 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:20.518 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:20.519 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=995), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [2ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [14ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [18ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [19ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [38ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [38ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [55ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [56ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [58ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [58ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [151ms] refreshing cache from master, [151ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [155ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [157ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [159ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [159ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [435ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [437ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [438ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [438ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [991ms] refreshing cache from master, [991ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [995ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@66720331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@134e0caa -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318) -> (continuation of Deferred@1877883845 after retry RPC@614449563) -> (continuation of Deferred@1339375715 after retry RPC@354210587), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2069e25d -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318) -> (continuation of Deferred@1877883845 after retry RPC@614449563) -> (continuation of Deferred@1339375715 after retry RPC@354210587))) after lookup
08:01:20.520 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:20.521 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:20.521 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:20.521 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:20.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.688413  5289 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:01:20.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.688854  5289 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.691387  5289 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:20.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:20.695325  5164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:20.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:20.699698  5164 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:20.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.703305  5048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da" is_pre_election: true
08:01:20.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.704911  5162 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea; no voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da
08:01:20.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.705701  5289 raft_consensus.cc:2749] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
08:01:20.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.951992  5294 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:20.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.952428  5294 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:20.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.954979  5294 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:20.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:20.958771  4978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:20.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:20.964098  4978 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:20.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.968354  5232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" is_pre_election: true
08:01:20.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.968991  5232 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 3.
08:01:20.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.970356  4976 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:20.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.971041  5294 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 4 FOLLOWER]: Leader pre-election won for term 5
08:01:20.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.971381  5294 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:20.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.971619  5294 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 4 FOLLOWER]: Advancing to term 5
08:01:20.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.975988  5294 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:20.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.977610  5294 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 5 election: Requested vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:20.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.978729  5232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:20.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.979219  5232 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 3 FOLLOWER]: Advancing to term 5
08:01:20.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:20.980950  4978 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:20.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.986049  5232 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 5.
08:01:20.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.987138  4976 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:20.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.987774  5294 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 5 FOLLOWER]: Leader election won for term 5
08:01:20.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.989391  5294 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 5 LEADER]: Becoming Leader. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Running, Role: LEADER
08:01:20.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:20.990278  5294 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:21.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:21.000319  4141 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da reported cstate change: term changed from 4 to 5, leader changed from 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131) to cb31ba83425d4f48bbc683abe7c675da (127.3.217.130). New cstate: current_term: 5 leader_uuid: "cb31ba83425d4f48bbc683abe7c675da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: HEALTHY } } }
08:01:21.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:21.553711  5232 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 5 FOLLOWER]: Refusing update from remote peer cb31ba83425d4f48bbc683abe7c675da: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
08:01:21.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:21.555313  5313 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
08:01:21.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:21.604934  4978 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:23.279 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:23.279 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:23.280 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=3756), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [2ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [14ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [18ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [19ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [38ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [38ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [51ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [55ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [56ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [58ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [58ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [151ms] refreshing cache from master, [151ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [155ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [157ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [159ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [159ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [435ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [437ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [438ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [438ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [991ms] refreshing cache from master, [991ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [995ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [997ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [998ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [998ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [3751ms] refreshing cache from master, [3751ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [3755ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@66720331(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@134e0caa -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318) -> (continuation of Deferred@1877883845 after retry RPC@614449563) -> (continuation of Deferred@1339375715 after retry RPC@354210587) -> (continuation of Deferred@128965568 after retry RPC@1951941669), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2069e25d -> wakeup thread Time-limited test -> (continuation of Deferred@823807076 after retry RPC@1577986832) -> (continuation of Deferred@934597972 after retry RPC@407711257) -> (continuation of Deferred@865988850 after retry RPC@1112616318) -> (continuation of Deferred@1877883845 after retry RPC@614449563) -> (continuation of Deferred@1339375715 after retry RPC@354210587) -> (continuation of Deferred@128965568 after retry RPC@1951941669))) after lookup
08:01:23.281 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:23.282 [DEBUG - kudu-nio-1] (Connection.java:757) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] Successfully connected to peer
08:01:23.300 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xd197b240, L:/127.0.0.1:60266 - R:/127.3.217.130:44033] using TOKEN/null
08:01:23.352 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.131:33863
08:01:23.353 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:23.354298  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:23.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:23.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:23.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.695940  5338 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:23.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.696399  5338 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:23.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.696844  5338 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:23.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.730190  5338 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:23.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.731140  5338 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.131
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:23.768698  5338 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:23.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:23.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:23.770090  5338 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:23.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:23.771667  5338 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:23.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.784312  5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:23.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.967918  4978 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:01:25.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:23.785748  5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:25.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:25.071858  5347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:25.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:25.074074  5346 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1283 milliseconds
08:01:25.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.074148  5338 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:25.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.075397  5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:25.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.078473  5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:25.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.079900  5338 hybrid_clock.cc:648] HybridClock initialized: now 1783497685079845 us; error 62 us; skew 500 ppm
08:01:25.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.080812  5338 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.086704  5338 webserver.cc:533] Webserver started at http://127.3.217.131:39685/ using document root <none> and password file <none>
08:01:25.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.087642  5338 fs_manager.cc:362] Metadata directory not provided
08:01:25.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.087883  5338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:25.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.095844  5338 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
08:01:25.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.102257  5355 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:25.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.103415  5338 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
08:01:25.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.103788  5338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:25.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:25.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:02 on dist-test-slave-d17k"
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.105789  5338 fs_report.cc:389] FS layout report
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/data
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:25.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:25.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:25.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.163812  5338 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:25.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.165516  5338 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:25.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.166106  5338 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:25.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.168797  5338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:25.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.175230  5362 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:25.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.183568  5338 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:25.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.183913  5338 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
08:01:25.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.184280  5338 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:25.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.188992  5338 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:25.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.189242  5338 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.003s	sys 0.003s
08:01:25.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.189728  5362 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap starting.
08:01:25.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.251381  5362 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Log is configured to *not* fsync() on all Append() calls
08:01:25.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.313098  5362 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:25.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.314106  5362 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap complete.
08:01:25.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.315912  5362 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent bootstrapping tablet: real 0.127s	user 0.088s	sys 0.036s
08:01:25.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.340226  5362 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:25.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.344262  5362 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Initialized, Role: FOLLOWER
08:01:25.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.345713  5362 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:25.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.350136  5362 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent starting tablet: real 0.034s	user 0.028s	sys 0.006s
08:01:25.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.367955  5338 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.131:33863
08:01:25.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.368213  5470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.131:33863 every 8 connection(s)
08:01:25.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.370689  5338 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:25.381 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:25.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.380254  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 5338
08:01:25.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.446959  5473 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:25.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.447631  5473 heartbeater.cc:461] Registering TS with master...
08:01:25.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.449312  5473 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:25.454 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:25.455 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.130:44033
08:01:25.455 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:01:25.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.455689  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 4955
08:01:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.453226  5471 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.458700  5471 heartbeater.cc:461] Registering TS with master...
08:01:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.459647  5471 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.460057  3987 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.454068  5472 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.461541  5472 heartbeater.cc:461] Registering TS with master...
08:01:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.462406  5472 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:25.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.465301  4061 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:25.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.465556  4141 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:25.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:25.468248  4141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.131:58057
08:01:25.489 [DEBUG - kudu-nio-1] (Connection.java:239) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] handling channelInactive
08:01:25.489 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:25.489 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state READY due to: connection closed
08:01:25.490 [DEBUG - Time-limited test] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:25.491 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.491 [DEBUG - kudu-nio-2] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.492 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:25.498 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:25.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@397467173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@53d1e627 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b466419 -> wakeup thread Time-limited test)) after lookup
08:01:25.499 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:25.500 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.500 [DEBUG - kudu-nio-3] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.500 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.518 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:25.519 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:25.519 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [10ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [11ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [12ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@397467173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@53d1e627 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b466419 -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660))) after lookup
08:01:25.520 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:25.521 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.521 [DEBUG - kudu-nio-4] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.521 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.558 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:25.559 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:25.559 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [10ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [11ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [12ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [32ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [32ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@397467173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@53d1e627 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b466419 -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256))) after lookup
08:01:25.560 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:25.561 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.561 [DEBUG - kudu-nio-5] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.561 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:25.698 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:25.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=209), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [10ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [11ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [12ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [32ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [32ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [71ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [72ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [73ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [205ms] refreshing cache from master, [205ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [208ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@397467173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@53d1e627 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256) -> (continuation of Deferred@561192182 after retry RPC@425187749), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b466419 -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256) -> (continuation of Deferred@561192182 after retry RPC@425187749))) after lookup
08:01:25.699 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:25.701 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.701 [DEBUG - kudu-nio-6] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:25.701 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:26.018 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:26.019 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:26.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=529), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [10ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [11ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [12ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [32ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [32ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [71ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [72ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [73ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [205ms] refreshing cache from master, [205ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [208ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [210ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [212ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [212ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [525ms] refreshing cache from master, [525ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [529ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@397467173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@53d1e627 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256) -> (continuation of Deferred@561192182 after retry RPC@425187749) -> (continuation of Deferred@1551884945 after retry RPC@2076344906), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b466419 -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256) -> (continuation of Deferred@561192182 after retry RPC@425187749) -> (continuation of Deferred@1551884945 after retry RPC@2076344906))) after lookup
08:01:26.020 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:26.021 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:26.021 [DEBUG - kudu-nio-7] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:26.021 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:26.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.471243  5472 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:26.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.612843  5482 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cb31ba83425d4f48bbc683abe7c675da)
08:01:26.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.613339  5482 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:26.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.615307  5482 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:26.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:26.619598  5162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:26.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:26.623788  5162 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:26.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.628113  5425 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "802ab38827f94c91a1d169aa96b11b7d" is_pre_election: true
08:01:26.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.628820  5425 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee4f3ebf8a7415ca003f30ddf138eea in term 4.
08:01:26.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.630095  5164 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, 802ab38827f94c91a1d169aa96b11b7d; no voters: cb31ba83425d4f48bbc683abe7c675da
08:01:26.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.630774  5482 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 5 FOLLOWER]: Leader pre-election won for term 6
08:01:26.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.631153  5482 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 5 FOLLOWER]: Starting leader election (detected failure of leader cb31ba83425d4f48bbc683abe7c675da)
08:01:26.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.631409  5482 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 5 FOLLOWER]: Advancing to term 6
08:01:26.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.638248  5482 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:26.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.640064  5482 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 6 election: Requested vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:26.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.640780  5425 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:26.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.641392  5425 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 4 FOLLOWER]: Advancing to term 6
08:01:26.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:26.643697  5162 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:26.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.648128  5425 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee4f3ebf8a7415ca003f30ddf138eea in term 6.
08:01:26.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.649214  5164 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, 802ab38827f94c91a1d169aa96b11b7d; no voters: cb31ba83425d4f48bbc683abe7c675da
08:01:26.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.649933  5482 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 6 FOLLOWER]: Leader election won for term 6
08:01:26.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.651149  5482 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 6 LEADER]: Becoming Leader. State: Replica: 5ee4f3ebf8a7415ca003f30ddf138eea, State: Running, Role: LEADER
08:01:26.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.652248  5482 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:26.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.660485  4141 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea reported cstate change: term changed from 5 to 6, leader changed from cb31ba83425d4f48bbc683abe7c675da (127.3.217.130) to 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129). New cstate: current_term: 6 leader_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: UNKNOWN } } }
08:01:26.859 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:26.859 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:26.860 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1370), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [2ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [10ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [11ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [12ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [32ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [32ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [71ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [72ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [73ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [205ms] refreshing cache from master, [205ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [208ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [210ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [212ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [212ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [525ms] refreshing cache from master, [525ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [529ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [531ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [532ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [532ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [1365ms] refreshing cache from master, [1366ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [1369ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@397467173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@53d1e627 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256) -> (continuation of Deferred@561192182 after retry RPC@425187749) -> (continuation of Deferred@1551884945 after retry RPC@2076344906) -> (continuation of Deferred@1013223538 after retry RPC@1406899558), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b466419 -> wakeup thread Time-limited test -> (continuation of Deferred@1358806110 after retry RPC@1216572660) -> (continuation of Deferred@1632004512 after retry RPC@846286256) -> (continuation of Deferred@561192182 after retry RPC@425187749) -> (continuation of Deferred@1551884945 after retry RPC@2076344906) -> (continuation of Deferred@1013223538 after retry RPC@1406899558))) after lookup
08:01:26.861 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:26.862 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] Successfully connected to peer
08:01:26.883 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x7718f3ce, L:/127.0.0.1:40130 - R:/127.3.217.129:36183] using TOKEN/null
08:01:26.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.901383  5425 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 6 FOLLOWER]: Refusing update from remote peer 5ee4f3ebf8a7415ca003f30ddf138eea: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
08:01:26.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.903136  5487 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [LEADER]: Connected to new peer: Peer: permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.001s
08:01:26.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:26.903729  5162 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea -> Peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Couldn't send request to peer cb31ba83425d4f48bbc683abe7c675da. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.919411  5504 mvcc.cc:204] Tried to move back new op lower bound from 7305206510830186496 to 7305206501344452608. Current Snapshot: MvccSnapshot[applied={T|T < 7305206510830186496}]
08:01:26.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.927477  5502 mvcc.cc:204] Tried to move back new op lower bound from 7305206525528481792 to 7305206524535382016. Current Snapshot: MvccSnapshot[applied={T|T < 7305206525528481792}]
08:01:26.983 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.130:44033
08:01:26.985 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:26.986837  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:26.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:44033
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43807
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:26.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:27.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:27.299161  5511 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:27.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:27.299624  5511 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:27.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:27.300172  5511 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:27.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:27.333381  5511 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:27.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:27.334291  5511 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.130
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:27.370594  5511 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:44033
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43807
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:01:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:27.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:27.371867  5511 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:27.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:27.373430  5511 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:27.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:27.387878  5520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:29.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:28.790010  5519 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 5511
08:01:29.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:29.200402  5511 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.806s	user 0.642s	sys 1.104s
08:01:29.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:29.201521  5511 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.808s	user 0.642s	sys 1.104s
08:01:29.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:27.394205  5521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:29.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:29.203509  5523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:29.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:29.206174  5522 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1807 milliseconds
08:01:29.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.206272  5511 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:29.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.207465  5511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:29.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.209590  5511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:29.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.210949  5511 hybrid_clock.cc:648] HybridClock initialized: now 1783497689210920 us; error 56 us; skew 500 ppm
08:01:29.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.211771  5511 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:29.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.218204  5511 webserver.cc:533] Webserver started at http://127.3.217.130:43807/ using document root <none> and password file <none>
08:01:29.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.219091  5511 fs_manager.cc:362] Metadata directory not provided
08:01:29.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.219287  5511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:29.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.226750  5511 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.004s
08:01:29.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.231722  5542 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:29.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.232801  5511 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.003s
08:01:29.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.233139  5511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:00 on dist-test-slave-d17k"
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.234966  5511 fs_report.cc:389] FS layout report
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/data
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:29.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:29.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.299007  5511 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:29.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.300654  5511 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:29.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.301224  5511 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:29.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.304164  5511 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:29.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.310483  5550 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:29.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.319753  5511 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:29.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.319993  5511 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
08:01:29.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.320232  5511 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:29.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.325160  5511 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:29.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.325376  5511 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.002s
08:01:29.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.325861  5550 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap starting.
08:01:29.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.408586  5550 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Log is configured to *not* fsync() on all Append() calls
08:01:29.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.512936  5550 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:29.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.514381  5550 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap complete.
08:01:29.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.516496  5550 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent bootstrapping tablet: real 0.191s	user 0.137s	sys 0.046s
08:01:29.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.525295  5511 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.130:44033
08:01:29.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.525424  5658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.130:44033 every 8 connection(s)
08:01:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.528949  5511 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:29.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.537509  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 5511
08:01:29.539 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:29.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.545720  5550 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:29.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.549589  5550 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Initialized, Role: FOLLOWER
08:01:29.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.550567  5550 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:29.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.570363  5550 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent starting tablet: real 0.053s	user 0.035s	sys 0.012s
08:01:29.580 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:29.581 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.129:36183
08:01:29.581 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:01:29.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.582347  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 5139
08:01:29.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.589268  5660 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:29.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.590260  5660 heartbeater.cc:461] Registering TS with master...
08:01:29.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.592106  5660 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:29.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.596122  5659 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:29.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.596503  5659 heartbeater.cc:461] Registering TS with master...
08:01:29.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.597132  4142 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:29.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.597564  5659 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:29.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.598623  5661 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:29.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.599028  5661 heartbeater.cc:461] Registering TS with master...
08:01:29.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.600035  5661 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:29.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.600705  4142 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.130:47367
08:01:29.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.601989  4062 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:29.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.604453  3987 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:29.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:29.605669  5660 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:29.621 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] handling channelInactive
08:01:29.621 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state READY due to: connection closed
08:01:29.622 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:29.623 [DEBUG - Time-limited test] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:29.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:29.622634  5672 negotiation.cc:336] Failed RPC negotiation. Trace:
08:01:29.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:29.569179 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.3.217.129:41053 (local address 127.3.217.130:44033)
08:01:29.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:29.576161 (+  6982us) server_negotiation.cc:207] Beginning negotiation
08:01:29.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:29.576189 (+    28us) server_negotiation.cc:400] Waiting for connection header
08:01:29.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:29.582377 (+  6188us) server_negotiation.cc:408] Connection header received
08:01:29.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:29.620934 (+ 38557us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.129:41053: BlockingRecv error: recv got EOF from 127.3.217.129:41053 (error 108)
08:01:29.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":6273,"thread_start_us":6824,"threads_started":1}
08:01:29.624 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.624 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.625 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.638 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:29.639 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:29.639 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@692130525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@26bec236 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3de16dc7 -> wakeup thread Time-limited test)) after lookup
08:01:29.640 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:29.641 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.641 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.641 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:29.658 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:29.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [18ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [18ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@692130525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@26bec236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3de16dc7 -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402))) after lookup
08:01:29.659 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:29.660 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.660 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.660 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:29.698 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:29.698 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [18ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [18ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [37ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [37ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@692130525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@26bec236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3de16dc7 -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276))) after lookup
08:01:29.699 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:29.700 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.700 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.700 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.838 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:29.838 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:29.839 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=215), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [18ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [18ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [37ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [37ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [76ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [77ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [77ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [215ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@692130525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@26bec236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276) -> (continuation of Deferred@478196803 after retry RPC@1472462350), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3de16dc7 -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276) -> (continuation of Deferred@478196803 after retry RPC@1472462350))) after lookup
08:01:29.839 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:29.841 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.841 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:29.841 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:30.298 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:30.298 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:30.299 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=676), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [18ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [18ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [37ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [37ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [76ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [77ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [77ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [215ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [216ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [218ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [218ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [670ms] refreshing cache from master, [671ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [675ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@692130525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@26bec236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276) -> (continuation of Deferred@478196803 after retry RPC@1472462350) -> (continuation of Deferred@1788302196 after retry RPC@585950118), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3de16dc7 -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276) -> (continuation of Deferred@478196803 after retry RPC@1472462350) -> (continuation of Deferred@1788302196 after retry RPC@585950118))) after lookup
08:01:30.300 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:30.301 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:30.301 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:30.301 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:30.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.794117  5673 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4f3ebf8a7415ca003f30ddf138eea)
08:01:30.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.794564  5673 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:30.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.797155  5673 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:30.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:30.801771  5356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:30.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:30.806439  5356 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:30.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.811288  5613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da" is_pre_election: true
08:01:30.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.812109  5613 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 5.
08:01:30.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.813573  5356 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da; no voters: 5ee4f3ebf8a7415ca003f30ddf138eea
08:01:30.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.814318  5673 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 6 FOLLOWER]: Leader pre-election won for term 7
08:01:30.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.814682  5673 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5ee4f3ebf8a7415ca003f30ddf138eea)
08:01:30.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.814918  5673 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 6 FOLLOWER]: Advancing to term 7
08:01:30.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.819239  5673 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:30.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.820991  5673 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 7 election: Requested vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:30.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.822593  5613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:30.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.823109  5613 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 5 FOLLOWER]: Advancing to term 7
08:01:30.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:30.825322  5356 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:30.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.830130  5613 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 7.
08:01:30.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.831004  5356 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da; no voters: 5ee4f3ebf8a7415ca003f30ddf138eea
08:01:30.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.831604  5673 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 7 FOLLOWER]: Leader election won for term 7
08:01:30.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.833258  5673 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 7 LEADER]: Becoming Leader. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Running, Role: LEADER
08:01:30.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.834113  5673 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:30.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:30.843946  4142 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d reported cstate change: term changed from 6 to 7, leader changed from 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129) to 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131). New cstate: current_term: 7 leader_uuid: "802ab38827f94c91a1d169aa96b11b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: UNKNOWN } } }
08:01:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:31.220245  5613 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 7 FOLLOWER]: Refusing update from remote peer 802ab38827f94c91a1d169aa96b11b7d: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
08:01:31.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:31.221776  5673 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
08:01:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.253808  5356 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d -> Peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Couldn't send request to peer 5ee4f3ebf8a7415ca003f30ddf138eea. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:31.260 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:31.260 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:31.261 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1638), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [2ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [2ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [15ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [16ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [18ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [18ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [35ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [36ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [37ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [37ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [76ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [77ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [77ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [215ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [216ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [218ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [218ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [670ms] refreshing cache from master, [671ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [675ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [676ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [678ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [678ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1631ms] refreshing cache from master, [1631ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [1637ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@692130525(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@26bec236 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276) -> (continuation of Deferred@478196803 after retry RPC@1472462350) -> (continuation of Deferred@1788302196 after retry RPC@585950118) -> (continuation of Deferred@1197785914 after retry RPC@2082301718), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3de16dc7 -> wakeup thread Time-limited test -> (continuation of Deferred@1816060899 after retry RPC@742502402) -> (continuation of Deferred@1340393922 after retry RPC@1845797276) -> (continuation of Deferred@478196803 after retry RPC@1472462350) -> (continuation of Deferred@1788302196 after retry RPC@585950118) -> (continuation of Deferred@1197785914 after retry RPC@2082301718))) after lookup
08:01:31.261 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:31.262 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] Successfully connected to peer
08:01:31.296 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x86f13673, L:/127.0.0.1:44784 - R:/127.3.217.131:33863] using TOKEN/null
08:01:31.340 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.129:36183
08:01:31.340 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:01:31.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:31.342188  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:31.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:31.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:36183
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45893
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:31.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:31.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.677644  5700 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:31.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.678103  5700 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:31.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.678541  5700 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:31.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.711514  5700 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:31.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.712334  5700 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.129
08:01:31.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:31.753420  5700 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:36183
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45893
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:31.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:31.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:31.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:31.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:31.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:31.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:31.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:31.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:31.754814  5700 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:31.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:31.756481  5700 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:31.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.769806  5706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:33.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:33.172353  5705 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 5700
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:33.465299  5705 kernel_stack_watchdog.cc:198] Thread 5700 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:33.465431  5700 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.696s	user 0.572s	sys 1.004s
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:33.465770  5700 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.696s	user 0.572s	sys 1.004s
08:01:33.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:31.770615  5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:33.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:33.467104  5708 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1695 milliseconds
08:01:33.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.468253  5700 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:33.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:33.468353  5709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:33.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.471356  5700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:33.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.473623  5700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:33.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.475006  5700 hybrid_clock.cc:648] HybridClock initialized: now 1783497693474968 us; error 36 us; skew 500 ppm
08:01:33.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.475991  5700 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:33.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.483209  5700 webserver.cc:533] Webserver started at http://127.3.217.129:45893/ using document root <none> and password file <none>
08:01:33.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.484124  5700 fs_manager.cc:362] Metadata directory not provided
08:01:33.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.484326  5700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:33.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.492273  5700 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
08:01:33.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.497260  5722 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:33.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.498380  5700 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.498703  5700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:57 on dist-test-slave-d17k"
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.500972  5700 fs_report.cc:389] FS layout report
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/data
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:33.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:33.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.573779  5700 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:33.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.575433  5700 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:33.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.575845  5700 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:33.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.578807  5700 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:33.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.584758  5729 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.593509  5700 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.593748  5700 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
08:01:33.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.594031  5700 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:33.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.598992  5700 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:33.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.599190  5700 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
08:01:33.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.599665  5729 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap starting.
08:01:33.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.676373  5729 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Log is configured to *not* fsync() on all Append() calls
08:01:33.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.773579  5729 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:33.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.774677  5729 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap complete.
08:01:33.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.776880  5729 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent bootstrapping tablet: real 0.178s	user 0.123s	sys 0.044s
08:01:33.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.795285  5700 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.129:36183
08:01:33.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.795567  5837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.129:36183 every 8 connection(s)
08:01:33.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.793998  5729 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:33.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.797152  5729 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4f3ebf8a7415ca003f30ddf138eea, State: Initialized, Role: FOLLOWER
08:01:33.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.798168  5729 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:33.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.799319  5700 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:33.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.802009  5729 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent starting tablet: real 0.025s	user 0.023s	sys 0.001s
08:01:33.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.801993  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 5700
08:01:33.803 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:33.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.863327  5839 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:33.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.863862  5839 heartbeater.cc:461] Registering TS with master...
08:01:33.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.865433  5839 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:33.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.870569  4062 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:33.896 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:33.898 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.131:33863
08:01:33.898 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:33.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.898753  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 5338
08:01:33.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.914750  5840 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:33.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.915503  5840 heartbeater.cc:461] Registering TS with master...
08:01:33.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.916460  5840 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:33.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.919862  4141 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:33.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.923334  4141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.129:38327
08:01:33.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.924046  5844 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:33.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.924434  5844 heartbeater.cc:461] Registering TS with master...
08:01:33.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.925410  5844 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:33.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:33.930078  3986 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:33.942 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] handling channelInactive
08:01:33.942 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state READY due to: connection closed
08:01:33.943 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:33.943406  5850 negotiation.cc:336] Failed RPC negotiation. Trace:
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.879093 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.3.217.131:47689 (local address 127.3.217.129:36183)
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.890736 (+ 11643us) server_negotiation.cc:207] Beginning negotiation
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.890780 (+    44us) server_negotiation.cc:400] Waiting for connection header
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.894652 (+  3872us) server_negotiation.cc:408] Connection header received
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.895780 (+  1128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.895818 (+    38us) server_negotiation.cc:462] Received NEGOTIATE request from client
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.896216 (+   398us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.896443 (+   227us) server_negotiation.cc:227] Negotiated authn=SASL
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:33.941552 (+ 45109us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.131:47689: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:01:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10699,"thread_start_us":10352,"threads_started":1}
08:01:33.944 [DEBUG - Time-limited test] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:33.945 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.945 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.945 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.957 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:33.958 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:33.958 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [13ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1103150992(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@56f25569 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@56019271 -> wakeup thread Time-limited test)) after lookup
08:01:33.958 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:33.959 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.959 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.959 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.978 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:33.978 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:33.979 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [13ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [14ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [15ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [16ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1103150992(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@56f25569 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@56019271 -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923))) after lookup
08:01:33.980 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:33.981 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.981 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:33.981 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.018 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:34.019 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:34.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [13ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [14ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [15ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [16ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [35ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [37ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [37ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1103150992(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@56f25569 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@56019271 -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830))) after lookup
08:01:34.019 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:34.021 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.021 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.021 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.038 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:34.039 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:34.039 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=95), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [13ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [14ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [15ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [16ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [35ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [37ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [37ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [75ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [77ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [77ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [94ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1103150992(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@56f25569 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830) -> (continuation of Deferred@1617978402 after retry RPC@747648299), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@56019271 -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830) -> (continuation of Deferred@1617978402 after retry RPC@747648299))) after lookup
08:01:34.040 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:34.041 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.041 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.041 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.498 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:34.499 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:34.499 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=555), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [13ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [14ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [15ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [16ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [35ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [37ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [37ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [75ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [77ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [77ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [94ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [96ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [97ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [97ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [550ms] refreshing cache from master, [550ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [554ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1103150992(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@56f25569 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830) -> (continuation of Deferred@1617978402 after retry RPC@747648299) -> (continuation of Deferred@66858498 after retry RPC@778598971), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@56019271 -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830) -> (continuation of Deferred@1617978402 after retry RPC@747648299) -> (continuation of Deferred@66858498 after retry RPC@778598971))) after lookup
08:01:34.500 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:34.501 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.501 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.501 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:34.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.928552  5840 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:34.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.955595  5851 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:34.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.955997  5851 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:34.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.958293  5851 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:34.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:34.961431  5546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:34.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:34.965292  5546 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:34.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.969991  5793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" is_pre_election: true
08:01:34.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.970662  5793 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 6.
08:01:34.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.971926  5544 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:34.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.972700  5851 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 7 FOLLOWER]: Leader pre-election won for term 8
08:01:34.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.973021  5851 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:34.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.973361  5851 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 7 FOLLOWER]: Advancing to term 8
08:01:34.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.978017  5851 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.979630  5851 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 8 election: Requested vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:34.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.980849  5793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:34.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.981429  5793 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 6 FOLLOWER]: Advancing to term 8
08:01:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:34.983356  5546 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:34.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.988129  5793 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 8.
08:01:34.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.988977  5544 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:34.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.989630  5851 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 8 FOLLOWER]: Leader election won for term 8
08:01:34.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.991065  5851 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 8 LEADER]: Becoming Leader. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Running, Role: LEADER
08:01:34.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:34.991816  5851 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:35.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:35.000483  4141 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da reported cstate change: term changed from 7 to 8, leader changed from 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131) to cb31ba83425d4f48bbc683abe7c675da (127.3.217.130). New cstate: current_term: 8 leader_uuid: "cb31ba83425d4f48bbc683abe7c675da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: HEALTHY } } }
08:01:35.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:35.541083  5546 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:35.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:35.569809  5793 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 8 FOLLOWER]: Refusing update from remote peer cb31ba83425d4f48bbc683abe7c675da: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
08:01:35.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:35.571380  5869 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
08:01:36.398 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:36.398 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:36.399 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2455), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [13ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [14ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [15ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [16ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [34ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [35ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [37ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [37ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [74ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [75ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [77ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [77ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [94ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [96ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [97ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [97ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [550ms] refreshing cache from master, [550ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [554ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [555ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [557ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [557ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [2450ms] refreshing cache from master, [2450ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [2454ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1103150992(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@56f25569 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830) -> (continuation of Deferred@1617978402 after retry RPC@747648299) -> (continuation of Deferred@66858498 after retry RPC@778598971) -> (continuation of Deferred@635637867 after retry RPC@1219960654), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@56019271 -> wakeup thread Time-limited test -> (continuation of Deferred@282288898 after retry RPC@1251735923) -> (continuation of Deferred@1247286969 after retry RPC@1453539830) -> (continuation of Deferred@1617978402 after retry RPC@747648299) -> (continuation of Deferred@66858498 after retry RPC@778598971) -> (continuation of Deferred@635637867 after retry RPC@1219960654))) after lookup
08:01:36.399 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:36.400 [DEBUG - kudu-nio-6] (Connection.java:757) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] Successfully connected to peer
08:01:36.432 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x29aab054, L:/127.0.0.1:36594 - R:/127.3.217.130:44033] using TOKEN/null
08:01:36.482 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.131:33863
08:01:36.482 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:36.484122  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:36.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:36.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:36.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.848553  5888 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:36.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.849282  5888 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:36.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.850059  5888 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:36.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.890394  5888 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:36.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.891436  5888 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.131
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:36.929565  5888 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:36.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:36.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:36.930974  5888 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:36.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:36.932657  5888 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:36.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.947005  5895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:38.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:38.040720  5546 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:01:38.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.950196  5896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:38.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:36.951805  5898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:38.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:36.951313  5888 server_base.cc:1061] running on GCE node
08:01:38.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.234622  5888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:38.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.237417  5888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:38.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.238834  5888 hybrid_clock.cc:648] HybridClock initialized: now 1783497698238791 us; error 53 us; skew 500 ppm
08:01:38.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.239645  5888 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:38.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.245898  5888 webserver.cc:533] Webserver started at http://127.3.217.131:39685/ using document root <none> and password file <none>
08:01:38.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.246783  5888 fs_manager.cc:362] Metadata directory not provided
08:01:38.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.246978  5888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:38.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.255302  5888 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.001s	sys 0.004s
08:01:38.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.260450  5911 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:38.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.261677  5888 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:01:38.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.261967  5888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:02 on dist-test-slave-d17k"
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.263813  5888 fs_report.cc:389] FS layout report
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/data
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:38.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:38.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.339452  5888 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:38.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.341630  5888 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:38.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.342254  5888 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:38.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.345920  5888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.354071  5918 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:38.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.362428  5888 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:38.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.362757  5888 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
08:01:38.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.363106  5888 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:38.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.370662  5888 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:38.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.370966  5888 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.006s	sys 0.001s
08:01:38.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.371428  5918 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap starting.
08:01:38.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.461827  5918 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Log is configured to *not* fsync() on all Append() calls
08:01:38.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.597837  5888 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.131:33863
08:01:38.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.598018  6025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.131:33863 every 8 connection(s)
08:01:38.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.601326  5888 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:38.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.602707  5918 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:38.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.603758  5918 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap complete.
08:01:38.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.605599  5918 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent bootstrapping tablet: real 0.235s	user 0.170s	sys 0.050s
08:01:38.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.605496  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 5888
08:01:38.606 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:38.668 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:38.669 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.130:44033
08:01:38.669 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:01:38.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.670102  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 5511
08:01:38.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.700232  5918 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:38.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.704020  5918 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Initialized, Role: FOLLOWER
08:01:38.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.704936  5918 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:38.720 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:38.720 [DEBUG - kudu-nio-6] (Connection.java:239) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] handling channelInactive
08:01:38.720 [DEBUG - kudu-nio-6] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state READY due to: connection closed
08:01:38.721 [DEBUG - Time-limited test] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:38.722 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.722 [DEBUG - kudu-nio-7] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.723 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.732026  6027 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:38.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.732575  6027 heartbeater.cc:461] Registering TS with master...
08:01:38.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.734017  6027 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:38.739 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:38.739 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:38.739 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [3ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1161051283(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@35891614 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d0446b4 -> wakeup thread Time-limited test)) after lookup
08:01:38.740 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:38.741 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.741 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.741 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.737337  6026 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:38.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.745250  6026 heartbeater.cc:461] Registering TS with master...
08:01:38.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.746613  6026 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.753417  4142 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:38.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.757300  4142 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.131:47751
08:01:38.761 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:38.761 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:38.761 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [3ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [20ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [21ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [21ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1161051283(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@35891614 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d0446b4 -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845))) after lookup
08:01:38.762 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:38.768944  6036 negotiation.cc:336] Failed RPC negotiation. Trace:
08:01:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:38.676189 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.3.217.130:48817 (local address 127.3.217.131:33863)
08:01:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:38.700000 (+ 23811us) server_negotiation.cc:207] Beginning negotiation
08:01:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:38.700041 (+    41us) server_negotiation.cc:400] Waiting for connection header
08:01:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 08:01:38.733695 (+ 33654us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.130:48817: BlockingRecv error: recv got EOF from 127.3.217.130:48817 (error 108)
08:01:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3318,"thread_start_us":1764,"threads_started":1}
08:01:38.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.771414  6027 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.772228  5918 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent starting tablet: real 0.166s	user 0.042s	sys 0.041s
08:01:38.773 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.773 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.774 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.779356  4061 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:38.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.781404  6031 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:38.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.781728  6031 heartbeater.cc:461] Registering TS with master...
08:01:38.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.782364  6031 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:38.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:38.786468  3986 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:38.801 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:38.801 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:38.801 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [3ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [20ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [21ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [21ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [41ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [54ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [54ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1161051283(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@35891614 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d0446b4 -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148))) after lookup
08:01:38.802 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:38.803 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.803 [DEBUG - kudu-nio-2] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.804 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.938 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:38.938 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:38.938 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=218), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [3ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [20ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [21ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [21ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [41ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [54ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [54ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [82ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [84ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [84ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [214ms] refreshing cache from master, [214ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [217ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1161051283(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@35891614 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148) -> (continuation of Deferred@1758456083 after retry RPC@1500369062), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d0446b4 -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148) -> (continuation of Deferred@1758456083 after retry RPC@1500369062))) after lookup
08:01:38.939 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:38.941 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.941 [DEBUG - kudu-nio-3] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:38.941 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:39.218 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:39.218 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:39.219 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=498), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [3ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [20ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [21ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [21ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [41ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [54ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [54ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [82ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [84ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [84ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [214ms] refreshing cache from master, [214ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [217ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [219ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [221ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [221ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [494ms] refreshing cache from master, [495ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [498ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1161051283(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@35891614 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148) -> (continuation of Deferred@1758456083 after retry RPC@1500369062) -> (continuation of Deferred@710913659 after retry RPC@486585206), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d0446b4 -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148) -> (continuation of Deferred@1758456083 after retry RPC@1500369062) -> (continuation of Deferred@710913659 after retry RPC@486585206))) after lookup
08:01:39.220 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:39.221 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:39.221 [DEBUG - kudu-nio-4] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state CONNECTING due to: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:39.221 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033
08:01:40.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.036224  6042 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cb31ba83425d4f48bbc683abe7c675da)
08:01:40.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.036691  6042 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:40.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.039208  6042 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:40.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:40.044359  5723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:40.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:40.048822  5723 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:40.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.052296  5981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "802ab38827f94c91a1d169aa96b11b7d" is_pre_election: true
08:01:40.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.052939  5981 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee4f3ebf8a7415ca003f30ddf138eea in term 7.
08:01:40.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.054282  5725 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, 802ab38827f94c91a1d169aa96b11b7d; no voters: cb31ba83425d4f48bbc683abe7c675da
08:01:40.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.055028  6042 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 8 FOLLOWER]: Leader pre-election won for term 9
08:01:40.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.055413  6042 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 8 FOLLOWER]: Starting leader election (detected failure of leader cb31ba83425d4f48bbc683abe7c675da)
08:01:40.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.055814  6042 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 8 FOLLOWER]: Advancing to term 9
08:01:40.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.060129  6042 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:40.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.061622  6042 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 9 election: Requested vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:40.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.062454  5981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:40.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.063023  5981 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 7 FOLLOWER]: Advancing to term 9
08:01:40.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:40.065522  5723 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111)
08:01:40.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.069833  5981 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee4f3ebf8a7415ca003f30ddf138eea in term 9.
08:01:40.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.070731  5725 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, 802ab38827f94c91a1d169aa96b11b7d; no voters: cb31ba83425d4f48bbc683abe7c675da
08:01:40.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.071345  6042 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Leader election won for term 9
08:01:40.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.072916  6042 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 LEADER]: Becoming Leader. State: Replica: 5ee4f3ebf8a7415ca003f30ddf138eea, State: Running, Role: LEADER
08:01:40.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.074013  6042 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:40.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.082248  4141 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea reported cstate change: term changed from 8 to 9, leader changed from cb31ba83425d4f48bbc683abe7c675da (127.3.217.130) to 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129). New cstate: current_term: 9 leader_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: UNKNOWN } } }
08:01:40.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:40.474965  5723 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea -> Peer cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033): Couldn't send request to peer cb31ba83425d4f48bbc683abe7c675da. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.130:44033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:40.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.559374  5981 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 9 FOLLOWER]: Refusing update from remote peer 5ee4f3ebf8a7415ca003f30ddf138eea: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
08:01:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.560845  6042 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [LEADER]: Connected to new peer: Peer: permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
08:01:40.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:40.658 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:40.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1938), Traces: [0ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [3ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [3ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [20ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [21ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [21ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [40ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [41ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [54ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [54ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [80ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [82ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [84ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [84ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [214ms] refreshing cache from master, [214ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [217ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [219ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [221ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [221ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [494ms] refreshing cache from master, [495ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [498ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [499ms] sending RPC to server cb31ba83425d4f48bbc683abe7c675da, [501ms] received response from server cb31ba83425d4f48bbc683abe7c675da: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [501ms] delaying RPC due to: Network error: Failed to connect to peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033): Connection refused: /127.3.217.130:44033, [1934ms] refreshing cache from master, [1934ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [1937ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1161051283(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@35891614 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148) -> (continuation of Deferred@1758456083 after retry RPC@1500369062) -> (continuation of Deferred@710913659 after retry RPC@486585206) -> (continuation of Deferred@612876314 after retry RPC@1006918911), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d0446b4 -> wakeup thread Time-limited test -> (continuation of Deferred@351297993 after retry RPC@190755845) -> (continuation of Deferred@1541942792 after retry RPC@627591148) -> (continuation of Deferred@1758456083 after retry RPC@1500369062) -> (continuation of Deferred@710913659 after retry RPC@486585206) -> (continuation of Deferred@612876314 after retry RPC@1006918911))) after lookup
08:01:40.659 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:40.660 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] Successfully connected to peer
08:01:40.691 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0xb25ca493, L:/127.0.0.1:54620 - R:/127.3.217.129:36183] using TOKEN/null
08:01:40.746 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.130:44033
08:01:40.747 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:40.748107  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:40.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:44033
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43807
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:40.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:41.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:41.092861  6071 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:41.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:41.093536  6071 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:41.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:41.094100  6071 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:41.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:41.143996  6071 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:41.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:41.144866  6071 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.130
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:41.183887  6071 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:41.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.130:44033
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.130
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43807
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.130
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/logs
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:41.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:41.185283  6071 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:41.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:41.186849  6071 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:41.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:41.199901  6079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:42.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:41.201815  6080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:42.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:42.522010  6082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:42.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:42.524385  6081 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1318 milliseconds
08:01:42.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.524537  6071 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:42.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.526082  6071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:42.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.528553  6071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:42.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.530041  6071 hybrid_clock.cc:648] HybridClock initialized: now 1783497702529920 us; error 60 us; skew 500 ppm
08:01:42.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.530989  6071 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:42.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.538384  6071 webserver.cc:533] Webserver started at http://127.3.217.130:43807/ using document root <none> and password file <none>
08:01:42.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.539345  6071 fs_manager.cc:362] Metadata directory not provided
08:01:42.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.539600  6071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:42.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.547837  6071 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
08:01:42.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.552373  6094 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:42.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.553633  6071 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.553943  6071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:00 on dist-test-slave-d17k"
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.555858  6071 fs_report.cc:389] FS layout report
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/wal
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/data
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:42.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:42.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.608911  6071 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:42.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.610569  6071 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:42.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.610980  6071 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:42.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.613735  6071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:42.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.619539  6101 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:42.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.628489  6071 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:42.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.628779  6071 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.000s
08:01:42.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.629122  6071 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:42.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.637277  6071 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:42.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.637526  6071 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.007s	sys 0.000s
08:01:42.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.637923  6101 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap starting.
08:01:42.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.724387  6101 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Log is configured to *not* fsync() on all Append() calls
08:01:42.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.844457  6101 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:42.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.845688  6101 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Bootstrap complete.
08:01:42.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.847929  6101 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent bootstrapping tablet: real 0.210s	user 0.168s	sys 0.038s
08:01:42.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.862636  6071 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.130:44033
08:01:42.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.862764  6208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.130:44033 every 8 connection(s)
08:01:42.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.866639  6071 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-1/data/info.pb
08:01:42.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.870360  6101 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:42.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.873590  6101 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Initialized, Role: FOLLOWER
08:01:42.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.874728  6101 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:42.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.877359  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 6071
08:01:42.879 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:42.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.902599  6101 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da: Time spent starting tablet: real 0.054s	user 0.032s	sys 0.000s
08:01:42.914 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:42.915 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.129:36183
08:01:42.915 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:01:42.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.915952  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 5700
08:01:42.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.932659  6210 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:42.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.933459  6210 heartbeater.cc:461] Registering TS with master...
08:01:42.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.934762  6210 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:42.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.939358  4141 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:42.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.942334  4141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.130:42445
08:01:42.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.946334  6210 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:42.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.951098  6209 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:42.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.951472  6209 heartbeater.cc:461] Registering TS with master...
08:01:42.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.952147  6209 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:42.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.953472  6211 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:42.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.953833  6211 heartbeater.cc:461] Registering TS with master...
08:01:42.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.954475  6211 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:42.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.955871  4061 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:42.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:42.962280  3986 ts_manager.cc:194] Re-registered known tserver with Master: cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:42.968 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] handling channelInactive
08:01:42.968 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state READY due to: connection closed
08:01:42.968 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:42.969 [DEBUG - Time-limited test] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:42.970 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:42.970 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:42.970 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:42.978 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:42.978 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:42.978 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test)) after lookup
08:01:42.979 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:42.980 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:42.980 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:42.980 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:42.998 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:42.998 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:42.999 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [9ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [11ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905))) after lookup
08:01:42.999 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:43.000 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.001 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.001 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.038 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:43.039 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:43.039 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [9ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [11ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [32ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [32ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255))) after lookup
08:01:43.040 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:43.041 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.041 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.041 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.158 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:43.159 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:43.159 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [9ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [11ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [32ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [32ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [70ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [72ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [72ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [189ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430))) after lookup
08:01:43.160 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:43.161 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.161 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.161 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:43.658 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:43.658 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=689), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [9ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [11ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [32ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [32ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [70ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [72ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [72ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [189ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [190ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [192ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [192ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [685ms] refreshing cache from master, [685ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [688ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625))) after lookup
08:01:43.659 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:43.660 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.660 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.660 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.838 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:43.838 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:43.838 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=869), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [9ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [11ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [32ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [32ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [70ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [72ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [72ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [189ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [190ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [192ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [192ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [685ms] refreshing cache from master, [685ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [688ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [690ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [691ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [691ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [865ms] refreshing cache from master, [865ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [868ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625) -> (continuation of Deferred@1617117698 after retry RPC@455912342), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625) -> (continuation of Deferred@1617117698 after retry RPC@455912342))) after lookup
08:01:43.839 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:43.840 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.840 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.840 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.879 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:43.880 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)[L],802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:43.880 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=911), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [9ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [11ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [32ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [32ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [70ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [72ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [72ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [189ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [190ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [192ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [192ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [685ms] refreshing cache from master, [685ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [688ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [690ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [691ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [691ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [865ms] refreshing cache from master, [865ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [868ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [870ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [871ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [871ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [905ms] refreshing cache from master, [905ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [910ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625) -> (continuation of Deferred@1617117698 after retry RPC@455912342) -> (continuation of Deferred@476844975 after retry RPC@1650617263), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625) -> (continuation of Deferred@1617117698 after retry RPC@455912342) -> (continuation of Deferred@476844975 after retry RPC@1650617263))) after lookup
08:01:43.881 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] connecting to peer
08:01:43.882 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.882 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] cleaning up while in state CONNECTING due to: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.882 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183
08:01:43.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.978091  6221 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4f3ebf8a7415ca003f30ddf138eea)
08:01:43.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.978598  6221 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:43.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.981137  6221 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:43.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:43.984360  5912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:43.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:43.988473  5912 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:43.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.994654  6164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da" is_pre_election: true
08:01:43.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.995656  6164 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 8.
08:01:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.997022  5912 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da; no voters: 5ee4f3ebf8a7415ca003f30ddf138eea
08:01:43.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.997869  6221 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 9 FOLLOWER]: Leader pre-election won for term 10
08:01:43.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.998128  6221 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5ee4f3ebf8a7415ca003f30ddf138eea)
08:01:43.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:43.998349  6221 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 9 FOLLOWER]: Advancing to term 10
08:01:44.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.002728  6221 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:44.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.004101  6221 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 10 election: Requested vote from peers 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:44.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.005779  6164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "802ab38827f94c91a1d169aa96b11b7d" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da"
08:01:44.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.006299  6164 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 8 FOLLOWER]: Advancing to term 10
08:01:44.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:44.007598  5912 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111)
08:01:44.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.012970  6164 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802ab38827f94c91a1d169aa96b11b7d in term 10.
08:01:44.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.013861  5912 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da; no voters: 5ee4f3ebf8a7415ca003f30ddf138eea
08:01:44.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.014424  6221 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 10 FOLLOWER]: Leader election won for term 10
08:01:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.015527  6221 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 10 LEADER]: Becoming Leader. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Running, Role: LEADER
08:01:44.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.016413  6221 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:44.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.026226  4141 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d reported cstate change: term changed from 9 to 10, leader changed from 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129) to 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131). New cstate: current_term: 10 leader_uuid: "802ab38827f94c91a1d169aa96b11b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: UNKNOWN } } }
08:01:44.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:44.475709  5912 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d -> Peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Couldn't send request to peer 5ee4f3ebf8a7415ca003f30ddf138eea. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:44.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.529850  6164 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 10 FOLLOWER]: Refusing update from remote peer 802ab38827f94c91a1d169aa96b11b7d: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
08:01:44.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:44.531343  6221 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
08:01:44.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:44.704828  4173 debug-util.cc:398] Leaking SignalData structure 0x7b080009b160 after lost signal to thread 4104
08:01:44.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:44.705586  4173 debug-util.cc:398] Leaking SignalData structure 0x7b08000afc20 after lost signal to thread 4176
08:01:46.699 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:46.699 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:46.700 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=16, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=3730), Traces: [0ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [1ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [1ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [4ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [9ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [11ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [11ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [30ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [32ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [32ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [69ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [70ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [72ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [72ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [189ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [190ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [192ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [192ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [685ms] refreshing cache from master, [685ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [688ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [690ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [691ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [691ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [865ms] refreshing cache from master, [865ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [868ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [870ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [871ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [871ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [905ms] refreshing cache from master, [905ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [910ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [912ms] sending RPC to server 5ee4f3ebf8a7415ca003f30ddf138eea, [913ms] received response from server 5ee4f3ebf8a7415ca003f30ddf138eea: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [913ms] delaying RPC due to: Network error: Failed to connect to peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183): Connection refused: /127.3.217.129:36183, [3725ms] refreshing cache from master, [3725ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [3730ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@1256157049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@54223179 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625) -> (continuation of Deferred@1617117698 after retry RPC@455912342) -> (continuation of Deferred@476844975 after retry RPC@1650617263) -> (continuation of Deferred@571999811 after retry RPC@1406549692), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40b5791a -> wakeup thread Time-limited test -> (continuation of Deferred@1943691041 after retry RPC@416060905) -> (continuation of Deferred@551271840 after retry RPC@1547962255) -> (continuation of Deferred@2101991439 after retry RPC@30265430) -> (continuation of Deferred@929484582 after retry RPC@268343625) -> (continuation of Deferred@1617117698 after retry RPC@455912342) -> (continuation of Deferred@476844975 after retry RPC@1650617263) -> (continuation of Deferred@571999811 after retry RPC@1406549692))) after lookup
08:01:46.701 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:46.702 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] Successfully connected to peer
08:01:46.723 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x94b553a1, L:/127.0.0.1:56570 - R:/127.3.217.131:33863] using TOKEN/null
08:01:46.779 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.129:36183
08:01:46.779 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:01:46.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:46.780632  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:36183
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45893
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:47.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.029835  5912 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d -> Peer 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183): Couldn't send request to peer 5ee4f3ebf8a7415ca003f30ddf138eea. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.129:36183: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:01:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.134275  6260 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:47.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.134752  6260 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:47.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.135200  6260 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:47.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.168491  6260 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:47.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.169358  6260 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.129
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:47.206454  6260 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.129:36183
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.129
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45893
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:47.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.129
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/logs
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:47.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:47.207751  6260 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:47.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:47.210125  6260 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.227566  6270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:48.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.229715  6271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:48.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:47.231791  6273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:48.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:48.429391  6272 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
08:01:48.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.429500  6260 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:48.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.433494  6260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:48.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.435784  6260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:48.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.439307  6260 hybrid_clock.cc:648] HybridClock initialized: now 1783497708439260 us; error 51 us; skew 500 ppm
08:01:48.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.440207  6260 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:48.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.446925  6260 webserver.cc:533] Webserver started at http://127.3.217.129:45893/ using document root <none> and password file <none>
08:01:48.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.447844  6260 fs_manager.cc:362] Metadata directory not provided
08:01:48.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.448055  6260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:48.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.455964  6260 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.001s	sys 0.004s
08:01:48.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.460515  6286 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:48.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.461683  6260 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.462042  6260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:00:57 on dist-test-slave-d17k"
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.464025  6260 fs_report.cc:389] FS layout report
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/wal
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/data
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:48.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:48.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:48.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:48.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:48.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:48.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:48.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.511615  6260 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:48.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.513367  6260 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:48.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.513942  6260 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:48.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.516763  6260 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:48.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.522703  6293 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:48.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.531700  6260 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:48.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.531970  6260 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.001s
08:01:48.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.532305  6260 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:48.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.537266  6260 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:48.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.537485  6260 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.005s	sys 0.001s
08:01:48.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.537921  6293 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap starting.
08:01:48.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.619596  6293 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Log is configured to *not* fsync() on all Append() calls
08:01:48.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.739881  6260 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.129:36183
08:01:48.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.740046  6400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.129:36183 every 8 connection(s)
08:01:48.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.743196  6260 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-0/data/info.pb
08:01:48.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.754575  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 6260
08:01:48.756 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:48.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.776343  6293 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:48.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.777482  6293 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Bootstrap complete.
08:01:48.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.779299  6293 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent bootstrapping tablet: real 0.242s	user 0.162s	sys 0.059s
08:01:48.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.805706  6293 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:48.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.808869  6293 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4f3ebf8a7415ca003f30ddf138eea, State: Initialized, Role: FOLLOWER
08:01:48.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.810081  6293 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:48.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.820664  6293 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea: Time spent starting tablet: real 0.041s	user 0.024s	sys 0.012s
08:01:48.832 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:48.833 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.217.131:33863
08:01:48.834 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:48.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.834903  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 5888
08:01:48.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.842810  6401 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:48.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.843359  6401 heartbeater.cc:461] Registering TS with master...
08:01:48.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.844115  6403 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:48.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.844482  6403 heartbeater.cc:461] Registering TS with master...
08:01:48.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.845141  6401 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:48.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.845745  6403 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:48.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.850026  4062 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:48.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.850322  3986 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:48.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.858610  6402 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:48.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.858978  6402 heartbeater.cc:461] Registering TS with master...
08:01:48.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.860518  6402 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:48.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.863943  4142 ts_manager.cc:194] Re-registered known tserver with Master: 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:48.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.867129  4142 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.129:54723
08:01:48.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:48.871361  6402 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:48.898 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] handling channelInactive
08:01:48.898 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state READY due to: connection closed
08:01:48.906 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:48.907 [DEBUG - Time-limited test] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:48.908 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.908 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.908 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.918 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:48.919 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:48.919 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [11ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@600969276(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4113e924 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4b08ca7c -> wakeup thread Time-limited test)) after lookup
08:01:48.919 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:48.920 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.920 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.920 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.937 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:48.937 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:48.938 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [11ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [12ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [13ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [14ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@600969276(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4113e924 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4b08ca7c -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128))) after lookup
08:01:48.939 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:48.940 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.940 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.940 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.978 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:48.978 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:48.978 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [11ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [12ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [13ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [14ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [31ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [33ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [33ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [71ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@600969276(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4113e924 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4b08ca7c -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666))) after lookup
08:01:48.979 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:48.980 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.980 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:48.980 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.018 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:49.018 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:49.019 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=112), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [11ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [12ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [13ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [14ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [31ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [33ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [33ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [71ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [72ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [73ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [73ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@600969276(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4113e924 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4b08ca7c -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555))) after lookup
08:01:49.019 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:49.020 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.020 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.020 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.118 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:49.118 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:49.119 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=212), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [11ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [12ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [13ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [14ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [31ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [33ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [33ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [71ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [72ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [73ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [73ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [112ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [113ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [114ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [211ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@600969276(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4113e924 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555) -> (continuation of Deferred@1489999105 after retry RPC@1257604994), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4b08ca7c -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555) -> (continuation of Deferred@1489999105 after retry RPC@1257604994))) after lookup
08:01:49.119 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:49.120 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.120 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.120 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.798 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:49.798 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)[L],cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)]}]
08:01:49.799 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=892), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [11ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [12ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [13ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [14ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [31ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [33ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [33ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [71ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [72ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [73ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [73ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [112ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [113ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [114ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [211ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [212ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [213ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [213ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [887ms] refreshing cache from master, [887ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [890ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@600969276(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4113e924 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555) -> (continuation of Deferred@1489999105 after retry RPC@1257604994) -> (continuation of Deferred@750846472 after retry RPC@1248741765), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4b08ca7c -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555) -> (continuation of Deferred@1489999105 after retry RPC@1257604994) -> (continuation of Deferred@750846472 after retry RPC@1248741765))) after lookup
08:01:49.799 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] connecting to peer
08:01:49.800 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.801 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] cleaning up while in state CONNECTING due to: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:49.801 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863) for tablet bd21da974ee444b4adcfa3d96158740f: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863
08:01:50.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.123036  6418 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:01:50.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.123538  6418 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:50.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.126744  6418 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), cb31ba83425d4f48bbc683abe7c675da (127.3.217.130:44033)
08:01:50.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:50.133841  6289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:50.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:50.138095  6289 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:50.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.142251  6164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "cb31ba83425d4f48bbc683abe7c675da" is_pre_election: true
08:01:50.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.143658  6287 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea; no voters: 802ab38827f94c91a1d169aa96b11b7d, cb31ba83425d4f48bbc683abe7c675da
08:01:50.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.144483  6418 raft_consensus.cc:2749] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
08:01:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.192453  6423 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:50.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.192956  6423 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:50.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.196260  6423 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:50.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:50.199222  6097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:50.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:50.204267  6097 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:50.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.209358  6356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" is_pre_election: true
08:01:50.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.210073  6356 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 9.
08:01:50.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.211447  6095 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.212145  6423 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 10 FOLLOWER]: Leader pre-election won for term 11
08:01:50.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.212437  6423 raft_consensus.cc:493] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 10 FOLLOWER]: Starting leader election (detected failure of leader 802ab38827f94c91a1d169aa96b11b7d)
08:01:50.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.212774  6423 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 10 FOLLOWER]: Advancing to term 11
08:01:50.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.217900  6423 raft_consensus.cc:515] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:50.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.219733  6423 leader_election.cc:290] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 11 election: Requested vote from peers 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863), 5ee4f3ebf8a7415ca003f30ddf138eea (127.3.217.129:36183)
08:01:50.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.221030  6356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd21da974ee444b4adcfa3d96158740f" candidate_uuid: "cb31ba83425d4f48bbc683abe7c675da" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea"
08:01:50.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.221680  6356 raft_consensus.cc:3060] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 9 FOLLOWER]: Advancing to term 11
08:01:50.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:50.223862  6097 leader_election.cc:336] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111)
08:01:50.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.229540  6356 raft_consensus.cc:2468] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb31ba83425d4f48bbc683abe7c675da in term 11.
08:01:50.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.230737  6095 leader_election.cc:304] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ee4f3ebf8a7415ca003f30ddf138eea, cb31ba83425d4f48bbc683abe7c675da; no voters: 802ab38827f94c91a1d169aa96b11b7d
08:01:50.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.231663  6423 raft_consensus.cc:2804] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 11 FOLLOWER]: Leader election won for term 11
08:01:50.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.233299  6423 raft_consensus.cc:697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [term 11 LEADER]: Becoming Leader. State: Replica: cb31ba83425d4f48bbc683abe7c675da, State: Running, Role: LEADER
08:01:50.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.234174  6423 consensus_queue.cc:237] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.242630  4142 catalog_manager.cc:5697] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da reported cstate change: term changed from 10 to 11, leader changed from 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131) to cb31ba83425d4f48bbc683abe7c675da (127.3.217.130). New cstate: current_term: 11 leader_uuid: "cb31ba83425d4f48bbc683abe7c675da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } health_report { overall_health: HEALTHY } } }
08:01:50.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.616848  6356 raft_consensus.cc:1275] T bd21da974ee444b4adcfa3d96158740f P 5ee4f3ebf8a7415ca003f30ddf138eea [term 11 FOLLOWER]: Refusing update from remote peer cb31ba83425d4f48bbc683abe7c675da: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
08:01:50.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:50.618446  6423 consensus_queue.cc:1048] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
08:01:50.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:50.655354  6097 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:01:51.117 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet bd21da974ee444b4adcfa3d96158740f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297' with partition [<start>, <end>)
08:01:51.117 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=bd21da974ee444b4adcfa3d96158740f@[5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183),802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863),cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)[L]]}]
08:01:51.118 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=bd21da974ee444b4adcfa3d96158740f, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2211), Traces: [0ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [1ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [1ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [11ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [12ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [13ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [14ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [26ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [30ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [31ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [33ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [33ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [71ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [72ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [73ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [73ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [111ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [112ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [113ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [114ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [207ms] refreshing cache from master, [207ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [211ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [212ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [213ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [213ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [887ms] refreshing cache from master, [887ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [890ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, [892ms] sending RPC to server 802ab38827f94c91a1d169aa96b11b7d, [894ms] received response from server 802ab38827f94c91a1d169aa96b11b7d: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [894ms] delaying RPC due to: Network error: Failed to connect to peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863): Connection refused: /127.3.217.131:33863, [2206ms] refreshing cache from master, [2207ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.217.188:33435, [2210ms] Sub RPC GetTableLocations: received response from server master-127.3.217.188:33435: OK, deferred=Deferred@600969276(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007ff4443391e8@4113e924 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555) -> (continuation of Deferred@1489999105 after retry RPC@1257604994) -> (continuation of Deferred@750846472 after retry RPC@1248741765) -> (continuation of Deferred@1187417387 after retry RPC@2120013335), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4b08ca7c -> wakeup thread Time-limited test -> (continuation of Deferred@914661806 after retry RPC@1003673128) -> (continuation of Deferred@517675287 after retry RPC@857079666) -> (continuation of Deferred@264788834 after retry RPC@501344555) -> (continuation of Deferred@1489999105 after retry RPC@1257604994) -> (continuation of Deferred@750846472 after retry RPC@1248741765) -> (continuation of Deferred@1187417387 after retry RPC@2120013335))) after lookup
08:01:51.119 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] connecting to peer
08:01:51.120 [DEBUG - kudu-nio-6] (Connection.java:757) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] Successfully connected to peer
08:01:51.142 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x818756db, L:/127.0.0.1:49286 - R:/127.3.217.130:44033] using TOKEN/null
08:01:51.192 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.217.131:33863
08:01:51.193 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:01:51.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:51.194468  3942 external_mini_cluster.cc:1372] Running /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:51.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu
08:01:51.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:51.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:51.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:01:51.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:51.536274  6454 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:01:51.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:51.536913  6454 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:01:51.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:51.537631  6454 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:01:51.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:51.572134  6454 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:01:51.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:51.573225  6454 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.131
08:01:51.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:51.612318  6454 tablet_server_runner.cc:78] Tablet server non-default flags:
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.217.148:33727
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.217.131:33863
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.217.131
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39685
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.217.190:44599,127.3.217.189:36633,127.3.217.188:33435
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:01:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.217.131
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/logs
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 844bedc15e81bd62879377e001142786f7d931d8
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 07:43:22 UTC on e7f111948823
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 12771
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:01:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:51.614118  6454 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:01:51.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:51.615984  6454 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:01:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:51.628566  6461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:53.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:53.073314  6097 consensus_peers.cc:597] T bd21da974ee444b4adcfa3d96158740f P cb31ba83425d4f48bbc683abe7c675da -> Peer 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863): Couldn't send request to peer 802ab38827f94c91a1d169aa96b11b7d. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.131:33863: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:01:53.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:53.032187  6460 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 6454
08:01:53.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:53.148108  6454 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.519s	user 0.421s	sys 1.041s
08:01:53.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:51.631928  6462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:53.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:53.148610  6454 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.520s	user 0.421s	sys 1.041s
08:01:53.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:53.151010  6464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:01:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 08:01:53.154472  6463 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1524 milliseconds
08:01:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.154475  6454 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:01:53.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.155804  6454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:01:53.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.158015  6454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:01:53.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.159380  6454 hybrid_clock.cc:648] HybridClock initialized: now 1783497713159331 us; error 60 us; skew 500 ppm
08:01:53.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.160590  6454 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:01:53.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.168586  6454 webserver.cc:533] Webserver started at http://127.3.217.131:39685/ using document root <none> and password file <none>
08:01:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.169669  6454 fs_manager.cc:362] Metadata directory not provided
08:01:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.169883  6454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:01:53.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.178440  6454 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.002s	sys 0.002s
08:01:53.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.183468  6476 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:01:53.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.184715  6454 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.001s
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.185110  6454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data,/tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802ab38827f94c91a1d169aa96b11b7d"
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 08:01:02 on dist-test-slave-d17k"
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.187214  6454 fs_report.cc:389] FS layout report
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/wal
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/data
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:01:53.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:01:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:01:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:01:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:01:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:01:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:01:53.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.241444  6454 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:01:53.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.242980  6454 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:01:53.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.243404  6454 kserver.cc:163] Server-wide thread pool size limit: 3276
08:01:53.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.246054  6454 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:01:53.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.252199  6483 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:01:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.260809  6454 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:01:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.261114  6454 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
08:01:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.261452  6454 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:01:53.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.268687  6454 ts_tablet_manager.cc:616] Registered 1 tablets
08:01:53.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.268999  6454 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.004s	sys 0.003s
08:01:53.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.272971  6483 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap starting.
08:01:53.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.357697  6483 log.cc:826] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Log is configured to *not* fsync() on all Append() calls
08:01:53.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.476511  6454 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.131:33863
08:01:53.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.476619  6590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.217.131:33863 every 8 connection(s)
08:01:53.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.479398  6454 server_base.cc:1193] Dumped server information to /tmp/dist-test-task2tL2Mb/test-tmp/mini-kudu-cluster16154062897094670029/ts-2/data/info.pb
08:01:53.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.484208  3942 external_mini_cluster.cc:1434] Started /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu as pid 6454
08:01:53.485 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:01:53.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.510159  6483 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:01:53.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.511377  6483 tablet_bootstrap.cc:492] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Bootstrap complete.
08:01:53.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.513674  6483 ts_tablet_manager.cc:1403] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent bootstrapping tablet: real 0.241s	user 0.178s	sys 0.055s
08:01:53.538 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:53.556 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783497663297, tablet=bd21da974ee444b4adcfa3d96158740f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:01:53.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.557534  6483 raft_consensus.cc:359] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:53.562 [DEBUG - main] (Connection.java:575) [peer master-127.3.217.190:44599(127.3.217.190:44599)] disconnecting while in state READY
08:01:53.563 [DEBUG - main] (Connection.java:575) [peer master-127.3.217.189:36633(127.3.217.189:36633)] disconnecting while in state READY
08:01:53.563 [DEBUG - main] (Connection.java:575) [peer master-127.3.217.188:33435(127.3.217.188:33435)] disconnecting while in state READY
08:01:53.563 [DEBUG - main] (Connection.java:575) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] disconnecting while in state READY
08:01:53.564 [DEBUG - main] (Connection.java:575) [peer 5ee4f3ebf8a7415ca003f30ddf138eea(127.3.217.129:36183)] disconnecting while in state TERMINATED
08:01:53.564 [DEBUG - main] (Connection.java:575) [peer 802ab38827f94c91a1d169aa96b11b7d(127.3.217.131:33863)] disconnecting while in state TERMINATED
08:01:53.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.563736  6483 raft_consensus.cc:740] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802ab38827f94c91a1d169aa96b11b7d, State: Initialized, Role: FOLLOWER
08:01:53.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.564942  6483 consensus_queue.cc:260] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "802ab38827f94c91a1d169aa96b11b7d" member_type: VOTER last_known_addr { host: "127.3.217.131" port: 33863 } } peers { permanent_uuid: "5ee4f3ebf8a7415ca003f30ddf138eea" member_type: VOTER last_known_addr { host: "127.3.217.129" port: 36183 } } peers { permanent_uuid: "cb31ba83425d4f48bbc683abe7c675da" member_type: VOTER last_known_addr { host: "127.3.217.130" port: 44033 } }
08:01:53.566 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.3.217.189:36633(127.3.217.189:36633)] handling channelInactive
08:01:53.566 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.3.217.189:36633(127.3.217.189:36633)] cleaning up while in state READY due to: connection closed
08:01:53.566 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.3.217.190:44599(127.3.217.190:44599)] handling channelInactive
08:01:53.566 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.3.217.190:44599(127.3.217.190:44599)] cleaning up while in state READY due to: connection closed
08:01:53.568 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.3.217.188:33435(127.3.217.188:33435)] handling channelInactive
08:01:53.568 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.3.217.188:33435(127.3.217.188:33435)] cleaning up while in state READY due to: connection closed
08:01:53.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.569697  6483 ts_tablet_manager.cc:1434] T bd21da974ee444b4adcfa3d96158740f P 802ab38827f94c91a1d169aa96b11b7d: Time spent starting tablet: real 0.056s	user 0.029s	sys 0.003s
08:01:53.571 [DEBUG - kudu-nio-6] (AsyncKuduClient.java:2704) Releasing all remaining resources
08:01:53.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.570557  6591 heartbeater.cc:344] Connected to a master server at 127.3.217.189:36633
08:01:53.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.571127  6591 heartbeater.cc:461] Registering TS with master...
08:01:53.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.572669  6591 heartbeater.cc:507] Master 127.3.217.189:36633 requested a full tablet report, sending...
08:01:53.573 [DEBUG - kudu-nio-6] (Connection.java:239) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] handling channelInactive
08:01:53.573 [DEBUG - kudu-nio-6] (Connection.java:709) [peer cb31ba83425d4f48bbc683abe7c675da(127.3.217.130:44033)] cleaning up while in state READY due to: connection closed
08:01:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.574736  6592 heartbeater.cc:344] Connected to a master server at 127.3.217.188:33435
08:01:53.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.575106  6592 heartbeater.cc:461] Registering TS with master...
08:01:53.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.576500  6592 heartbeater.cc:507] Master 127.3.217.188:33435 requested a full tablet report, sending...
08:01:53.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.577761  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 6260
08:01:53.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.577979  4062 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:53.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.581853  4141 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:53.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.585531  4141 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.217.131:37305
08:01:53.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.592073  6592 heartbeater.cc:499] Master 127.3.217.188:33435 was elected leader, sending a full tablet report...
08:01:53.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.601958  6593 heartbeater.cc:344] Connected to a master server at 127.3.217.190:44599
08:01:53.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.602384  6593 heartbeater.cc:461] Registering TS with master...
08:01:53.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.603075  6593 heartbeater.cc:507] Master 127.3.217.190:44599 requested a full tablet report, sending...
08:01:53.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.607204  3987 ts_manager.cc:194] Re-registered known tserver with Master: 802ab38827f94c91a1d169aa96b11b7d (127.3.217.131:33863)
08:01:53.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.622463  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 6071
08:01:53.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.651980  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 6454
08:01:53.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.678596  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 3954
08:01:53.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.709578  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 4024
08:01:53.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 08:01:53.740562  3942 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task2tL2Mb/build/tsan/bin/kudu with pid 4099
08:01:54.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T08:01:54Z chronyd exiting

Time: 70.398

OK (1 test)

