JUnit version 4.13.2
.08:06:55.650 [INFO - main] (RandomUtils.java:49) Using random seed: 1786522015644
08:06:55.712 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
08:06:55.769 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
08:06:55.782 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskM4sYl4/test-tmp
08:06:55.785 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
08:06:55.789 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskM4sYl4/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
08:06:56.059 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

08:06:56.213 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:06:56.215 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

08:06:56.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.218504 13309 mini_oidc.cc:152] Starting JWKS server
08:06:56.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.221969 13309 webserver.cc:460] Webserver started at http://127.0.0.1:44807/ using document root <none> and password file <none>
08:06:56.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.222427 13309 mini_oidc.cc:189] Starting OIDC Discovery server
08:06:56.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.224287 13309 webserver.cc:460] Webserver started at http://127.0.0.1:37843/ using document root <none> and password file <none>
08:06:56.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
08:06:56.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:06:56Z Disabled control of system clock
08:06:56.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.280638 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:06:56.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:06:56.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/wal
08:06:56.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data
08:06:56.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:06:56.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:06:56.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:06:56.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:06:56.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:06:56.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:06:56.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/logs
08:06:56.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data/info.pb
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:06:56.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.126:39387
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.126
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:06:56.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:06:56.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:06:56.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:06:56.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:06:56.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:06:56.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:06:56.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44807/jwks.json with env {}
08:06:56.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.589542 13328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:06:56.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.590101 13328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:06:56.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.590679 13328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:06:56.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.621711 13328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
08:06:56.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.621999 13328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:06:56.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.622332 13328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
08:06:56.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.622637 13328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
08:06:56.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.622979 13328 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
08:06:56.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.623236 13328 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:44807/jwks.json
08:06:56.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.665380 13328 master_runner.cc:386] Master server non-default flags:
08:06:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:06:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:06:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:06:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:06:56.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data
08:06:56.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/wal
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.126:39387
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44807/jwks.json
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data/info.pb
08:06:56.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.126
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/logs
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:06:56.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:06:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:06:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:06:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:06:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:06:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:06:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.666673 13328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:06:56.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.668422 13328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:06:56.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.687525 13338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:06:57.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.687582 13341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:06:57.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:56.688787 13328 server_base.cc:1034] running on GCE node
08:06:57.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:56.687588 13337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:06:57.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.891680 13328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:06:57.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.894522 13328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:06:57.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.895921 13328 hybrid_clock.cc:648] HybridClock initialized: now 1786522017895820 us; error 116 us; skew 500 ppm
08:06:57.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.896798 13328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:06:57.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.907341 13328 webserver.cc:460] Webserver started at http://127.12.255.126:45131/ using document root <none> and password file <none>
08:06:57.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.908308 13328 fs_manager.cc:359] Metadata directory not provided
08:06:57.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.908499 13328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:06:57.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.908913 13328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
08:06:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.914075 13328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data/instance:
08:06:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec68a3ef4b364898a6daf77db56f0a93"
08:06:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:06:57 on dist-test-slave-ffrd"
08:06:57.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.915256 13328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/wal/instance:
08:06:57.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec68a3ef4b364898a6daf77db56f0a93"
08:06:57.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:06:57 on dist-test-slave-ffrd"
08:06:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.922730 13328 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.003s
08:06:57.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.928524 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:06:57.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.929840 13328 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
08:06:57.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.930132 13328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/wal
08:06:57.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec68a3ef4b364898a6daf77db56f0a93"
08:06:57.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:06:57 on dist-test-slave-ffrd"
08:06:57.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.930532 13328 fs_report.cc:383] FS layout report
08:06:57.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:06:57.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/wal
08:06:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/wal
08:06:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data/data
08:06:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:06:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:06:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:06:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:06:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:06:57.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:06:57.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:06:57.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:06:57.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:06:57.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:06:57.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:06:57.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.990433 13328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:06:57.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.991987 13328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:06:57.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:57.992774 13328 kserver.cc:163] Server-wide thread pool size limit: 3276
08:06:58.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.066845 13328 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.126:39387
08:06:58.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.066931 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.126:39387 every 8 connection(s)
08:06:58.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.069631 13328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/data/info.pb
08:06:58.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.072753 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 13328
08:06:58.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.073354 13309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-0/wal/instance
08:06:58.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.075248 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:06:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.076905 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:06:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:06:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/wal
08:06:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data
08:06:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:06:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:06:58.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/logs
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data/info.pb
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:06:58.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.125:46135
08:06:58.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.125
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:06:58.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:06:58.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44807/jwks.json with env {}
08:06:58.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.087360 13446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
08:06:58.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.119040 13446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } has no permanent_uuid. Determining permanent_uuid...
08:06:58.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.121793 13363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:58.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.125977 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.125:46135: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:58.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.175436 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } attempt: 1
08:06:58.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.179242 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.125:46135: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:58.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.247781 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } attempt: 2
08:06:58.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.252117 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.125:46135: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:58.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.343557 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } attempt: 3
08:06:58.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.347630 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.125:46135: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:58.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.401602 13449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:06:58.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.402256 13449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:06:58.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.402870 13449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:06:58.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.433552 13449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
08:06:58.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.433840 13449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:06:58.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.434103 13449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
08:06:58.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.434293 13449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
08:06:58.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.434484 13449 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
08:06:58.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.434624 13449 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:44807/jwks.json
08:06:58.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.469102 13449 master_runner.cc:386] Master server non-default flags:
08:06:58.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:06:58.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:06:58.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/wal
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.125:46135
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:06:58.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44807/jwks.json
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data/info.pb
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.125
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/logs
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:06:58.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:06:58.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:06:58.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:06:58.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.470449 13449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:06:58.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.472018 13449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:06:58.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.489753 13464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:06:58.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.491086 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } attempt: 4
08:06:58.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.495409 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.125:46135: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:58.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.794844 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } attempt: 5
08:06:58.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.799615 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.125:46135: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:59.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.347082 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } attempt: 6
08:06:59.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:59.354892 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.125:46135: Network error: Client connection negotiation failed: client connection to 127.12.255.125:46135: connect: Connection refused (error 111)
08:06:59.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.489766 13466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:06:59.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:58.489840 13469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:06:59.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:58.491439 13449 server_base.cc:1034] running on GCE node
08:06:59.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.695793 13449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:06:59.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.699102 13449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:06:59.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.700568 13449 hybrid_clock.cc:648] HybridClock initialized: now 1786522019700522 us; error 62 us; skew 500 ppm
08:06:59.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.701727 13449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:06:59.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.709393 13449 webserver.cc:460] Webserver started at http://127.12.255.125:37307/ using document root <none> and password file <none>
08:06:59.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.710577 13449 fs_manager.cc:359] Metadata directory not provided
08:06:59.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.710879 13449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:06:59.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.711526 13449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
08:06:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.717350 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data/instance:
08:06:59.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "18b2d0a700274e01940f91e1c6eb415e"
08:06:59.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:06:59 on dist-test-slave-ffrd"
08:06:59.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.718600 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/wal/instance:
08:06:59.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "18b2d0a700274e01940f91e1c6eb415e"
08:06:59.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:06:59 on dist-test-slave-ffrd"
08:06:59.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.726217 13449 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.003s	sys 0.005s
08:06:59.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.731933 13482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:06:59.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.733098 13449 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:06:59.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.733459 13449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/wal
08:06:59.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "18b2d0a700274e01940f91e1c6eb415e"
08:06:59.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:06:59 on dist-test-slave-ffrd"
08:06:59.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.733839 13449 fs_report.cc:383] FS layout report
08:06:59.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:06:59.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/wal
08:06:59.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/wal
08:06:59.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data/data
08:06:59.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:06:59.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:06:59.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:06:59.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:06:59.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:06:59.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:06:59.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:06:59.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:06:59.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:06:59.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:06:59.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:06:59.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.767910 13449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:06:59.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.769531 13449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:06:59.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.770085 13449 kserver.cc:163] Server-wide thread pool size limit: 3276
08:06:59.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.841585 13449 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.125:46135
08:06:59.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.841675 13566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.125:46135 every 8 connection(s)
08:06:59.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.844856 13449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/data/info.pb
08:06:59.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.850432 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:06:59.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.855254 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 13449
08:06:59.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.855628 13309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-1/wal/instance
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.857733 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/wal
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:06:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/logs
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data/info.pb
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:06:59.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.124:36715
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.124
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:06:59.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44807/jwks.json with env {}
08:06:59.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.865190 13570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
08:06:59.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.890568 13570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } has no permanent_uuid. Determining permanent_uuid...
08:06:59.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.910053 13570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } has no permanent_uuid. Determining permanent_uuid...
08:06:59.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:59.912388 13486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:06:59.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:59.916432 13570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:06:59.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:06:59.965942 13570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 1
08:06:59.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:06:59.969810 13570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.038215 13570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 2
08:07:00.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.042137 13570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.133589 13570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 3
08:07:00.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.137560 13570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.187007 13572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:00.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.187602 13572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:00.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.188190 13572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:00.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.220057 13572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
08:07:00.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.220366 13572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:00.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.220556 13572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
08:07:00.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.220737 13572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
08:07:00.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.220969 13572 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
08:07:00.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.221211 13572 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:44807/jwks.json
08:07:00.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.257102 13572 master_runner.cc:386] Master server non-default flags:
08:07:00.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/wal
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.124:36715
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44807/jwks.json
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data/info.pb
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.124
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:00.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/logs
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:00.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:00.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:00.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.258520 13572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:00.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.260743 13572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:00.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.279474 13598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:00.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.281026 13570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 4
08:07:00.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.285434 13570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.415445 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } attempt: 7
08:07:00.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.436012 13446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } has no permanent_uuid. Determining permanent_uuid...
08:07:00.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.442102 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.500762 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 1
08:07:00.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.504873 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.584923 13570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 5
08:07:00.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.586656 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 2
08:07:00.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.589926 13570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.591514 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.676898 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 3
08:07:00.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.681603 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:00.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:00.822048 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 4
08:07:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.828531 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:01.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.111986 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 5
08:07:01.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:01.115615 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:01.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.137336 13570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 6
08:07:01.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:01.144879 13570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:01.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.279533 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:01.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:00.279485 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:01.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:01.497828 13597 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
08:07:01.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.498073 13572 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:07:01.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.504735 13572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:01.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.508620 13572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:01.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.510131 13572 hybrid_clock.cc:648] HybridClock initialized: now 1786522021510048 us; error 107 us; skew 500 ppm
08:07:01.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.511782 13572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:01.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.522689 13572 webserver.cc:460] Webserver started at http://127.12.255.124:45627/ using document root <none> and password file <none>
08:07:01.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.524217 13572 fs_manager.cc:359] Metadata directory not provided
08:07:01.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.524545 13572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:01.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.525238 13572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
08:07:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.534054 13572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data/instance:
08:07:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a573591e58d84e75a303105cef6c901d"
08:07:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:01 on dist-test-slave-ffrd"
08:07:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.535897 13572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/wal/instance:
08:07:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a573591e58d84e75a303105cef6c901d"
08:07:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:01 on dist-test-slave-ffrd"
08:07:01.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.546983 13572 fs_manager.cc:691] Time spent creating directory manager: real 0.010s	user 0.008s	sys 0.004s
08:07:01.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.556373 13620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:01.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.558004 13572 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.004s
08:07:01.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.558482 13572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/wal
08:07:01.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a573591e58d84e75a303105cef6c901d"
08:07:01.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:01 on dist-test-slave-ffrd"
08:07:01.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.559031 13572 fs_report.cc:383] FS layout report
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/wal
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/wal
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data/data
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:01.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:01.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.634927 13572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:01.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.636437 13572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:01.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.636881 13572 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:01.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.668272 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 6
08:07:01.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:01.678834 13446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.255.124:36715: Network error: Client connection negotiation failed: client connection to 127.12.255.124:36715: connect: Connection refused (error 111)
08:07:01.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.728843 13572 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.124:36715
08:07:01.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.729086 13722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.124:36715 every 8 connection(s)
08:07:01.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.731995 13572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/data/info.pb
08:07:01.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.734576 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 13572
08:07:01.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.735111 13309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/master-2/wal/instance
08:07:01.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.739231 13723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:07:01.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.760576 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
08:07:01.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.798970 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } has no permanent_uuid. Determining permanent_uuid...
08:07:01.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.815191 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } has no permanent_uuid. Determining permanent_uuid...
08:07:01.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.835096 13723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d: Bootstrap starting.
08:07:01.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.843020 13723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d: Neither blocks nor log segments found. Creating new log.
08:07:01.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.845346 13723 log.cc:824] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d: Log is configured to *not* fsync() on all Append() calls
08:07:01.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.853819 13723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d: No bootstrap required, opened a new log
08:07:01.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.895967 13723 raft_consensus.cc:348] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:01.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.897044 13723 raft_consensus.cc:374] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:07:01.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.897389 13723 raft_consensus.cc:729] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a573591e58d84e75a303105cef6c901d, State: Initialized, Role: FOLLOWER
08:07:01.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.898739 13723 consensus_queue.cc:260] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:01.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.903340 13736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:01.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.904034 13736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: This master's current role is: FOLLOWER
08:07:01.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.905730 13723 sys_catalog.cc:564] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: configured and running, proceeding with master startup.
08:07:01.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:01.936506 13751 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:07:01.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:01.936811 13751 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:07:01.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.930575 13572 master_runner.cc:186] Error getting master registration for 127.12.255.126:39387: OK, instance_id { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" instance_seqno: 1786522018046748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:07:01.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:01.961090 13572 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
08:07:02.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:02.143338 13632 tablet.cc:2369] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:07:02.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.205385 13570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 7
08:07:02.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.226799 13570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: Bootstrap starting.
08:07:02.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.232694 13570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: Neither blocks nor log segments found. Creating new log.
08:07:02.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.234501 13570 log.cc:824] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: Log is configured to *not* fsync() on all Append() calls
08:07:02.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.238845 13736 raft_consensus.cc:482] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:02.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.239125 13570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: No bootstrap required, opened a new log
08:07:02.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.239372 13736 raft_consensus.cc:504] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:02.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.243073 13736 leader_election.cc:288] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec68a3ef4b364898a6daf77db56f0a93 (127.12.255.126:39387), 18b2d0a700274e01940f91e1c6eb415e (127.12.255.125:46135)
08:07:02.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.269212 13570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:02.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.270105 13570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:07:02.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.270377 13570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b2d0a700274e01940f91e1c6eb415e, State: Initialized, Role: FOLLOWER
08:07:02.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.271368 13570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:02.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.274890 13758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.275769 13758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [sys.catalog]: This master's current role is: FOLLOWER
08:07:02.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.276615 13570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [sys.catalog]: configured and running, proceeding with master startup.
08:07:02.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.276834 13524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a573591e58d84e75a303105cef6c901d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b2d0a700274e01940f91e1c6eb415e" is_pre_election: true
08:07:02.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.277801 13524 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a573591e58d84e75a303105cef6c901d in term 0.
08:07:02.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:02.280359 13490 tablet.cc:2369] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:07:02.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.280328 13626 leader_election.cc:302] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b2d0a700274e01940f91e1c6eb415e, a573591e58d84e75a303105cef6c901d; no voters: 
08:07:02.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.281350 13736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Leader pre-election won for term 1
08:07:02.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.281697 13736 raft_consensus.cc:482] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:07:02.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.282013 13736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 0 FOLLOWER]: Advancing to term 1
08:07:02.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.289952 13736 raft_consensus.cc:504] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:02.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.294705 13736 leader_election.cc:288] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [CANDIDATE]: Term 1 election: Requested vote from peers ec68a3ef4b364898a6daf77db56f0a93 (127.12.255.126:39387), 18b2d0a700274e01940f91e1c6eb415e (127.12.255.125:46135)
08:07:02.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.296046 13524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a573591e58d84e75a303105cef6c901d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b2d0a700274e01940f91e1c6eb415e"
08:07:02.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.297531 13524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [term 0 FOLLOWER]: Advancing to term 1
08:07:02.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.308445 13524 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a573591e58d84e75a303105cef6c901d in term 1.
08:07:02.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.310247 13626 leader_election.cc:302] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b2d0a700274e01940f91e1c6eb415e, a573591e58d84e75a303105cef6c901d; no voters: 
08:07:02.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.311769 13736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 1 FOLLOWER]: Leader election won for term 1
08:07:02.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.314112 13736 raft_consensus.cc:686] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [term 1 LEADER]: Becoming Leader. State: Replica: a573591e58d84e75a303105cef6c901d, State: Running, Role: LEADER
08:07:02.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.315863 13736 consensus_queue.cc:237] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:02.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.330474 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a573591e58d84e75a303105cef6c901d. Latest consensus state: current_term: 1 leader_uuid: "a573591e58d84e75a303105cef6c901d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.332757 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: This master's current role is: LEADER
08:07:02.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.342121 13779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
08:07:02.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.351370 13779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
08:07:02.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.383998 13524 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [term 1 FOLLOWER]: Refusing update from remote peer a573591e58d84e75a303105cef6c901d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:07:02.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.386224 13773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:07:02.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:02.390559 13780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:07:02.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:02.390898 13780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:07:02.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.361537 13449 master_runner.cc:186] Error getting master registration for 127.12.255.126:39387: OK, instance_id { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" instance_seqno: 1786522018046748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:07:02.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.430886 13449 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
08:07:02.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.458909 13758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a573591e58d84e75a303105cef6c901d. Latest consensus state: current_term: 1 leader_uuid: "a573591e58d84e75a303105cef6c901d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.460673 13758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [sys.catalog]: This master's current role is: FOLLOWER
08:07:02.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.466389 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a573591e58d84e75a303105cef6c901d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.467101 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: This master's current role is: LEADER
08:07:02.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.468698 13779 catalog_manager.cc:1347] Generated new cluster ID: dfe3e4072f4d4130bb99ea61b173a2f9
08:07:02.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.469175 13779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
08:07:02.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.487777 13758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a573591e58d84e75a303105cef6c901d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.491832 13758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e [sys.catalog]: This master's current role is: FOLLOWER
08:07:02.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.560267 13779 catalog_manager.cc:1370] Generated new certificate authority record
08:07:02.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.570048 13779 catalog_manager.cc:1504] Loading token signing keys...
08:07:02.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.654604 13779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d: Generated new TSK 0
08:07:02.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.656219 13779 catalog_manager.cc:1514] Initializing in-progress tserver states...
08:07:02.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.704620 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:02.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:0
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:02.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.712291 13446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } attempt: 7
08:07:02.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.740695 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93: Bootstrap starting.
08:07:02.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.747972 13446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93: Neither blocks nor log segments found. Creating new log.
08:07:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.750674 13446 log.cc:824] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93: Log is configured to *not* fsync() on all Append() calls
08:07:02.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.763116 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93: No bootstrap required, opened a new log
08:07:02.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.796855 13446 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:02.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.798120 13446 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:07:02.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.798617 13446 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec68a3ef4b364898a6daf77db56f0a93, State: Initialized, Role: FOLLOWER
08:07:02.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.799804 13446 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } }
08:07:02.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.826160 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.830138 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [sys.catalog]: This master's current role is: FOLLOWER
08:07:02.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.829749 13446 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [sys.catalog]: configured and running, proceeding with master startup.
08:07:02.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.830771 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a573591e58d84e75a303105cef6c901d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec68a3ef4b364898a6daf77db56f0a93"
08:07:02.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.833456 13409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a573591e58d84e75a303105cef6c901d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec68a3ef4b364898a6daf77db56f0a93" is_pre_election: true
08:07:02.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.830365 13407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [term 0 FOLLOWER]: Advancing to term 1
08:07:02.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.842847 13407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [term 1 FOLLOWER]: Refusing update from remote peer a573591e58d84e75a303105cef6c901d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:07:02.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.844714 13773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:07:02.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.929647 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a573591e58d84e75a303105cef6c901d. Latest consensus state: current_term: 1 leader_uuid: "a573591e58d84e75a303105cef6c901d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.931412 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [sys.catalog]: This master's current role is: FOLLOWER
08:07:02.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.932199 13808 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a573591e58d84e75a303105cef6c901d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:02.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:02.933723 13808 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93 [sys.catalog]: This master's current role is: FOLLOWER
08:07:03.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.009629 13835 catalog_manager.cc:1259] Loaded cluster ID: dfe3e4072f4d4130bb99ea61b173a2f9
08:07:03.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.010303 13835 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93: loading cluster ID for follower catalog manager: success
08:07:03.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.019328 13835 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93: acquiring CA information for follower catalog manager: success
08:07:03.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.034687 13835 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec68a3ef4b364898a6daf77db56f0a93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:07:03.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.396572 13780 catalog_manager.cc:1259] Loaded cluster ID: dfe3e4072f4d4130bb99ea61b173a2f9
08:07:03.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.398188 13780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: loading cluster ID for follower catalog manager: success
08:07:03.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.410085 13845 sys_catalog.cc:455] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a573591e58d84e75a303105cef6c901d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec68a3ef4b364898a6daf77db56f0a93" member_type: VOTER last_known_addr { host: "127.12.255.126" port: 39387 } } peers { permanent_uuid: "18b2d0a700274e01940f91e1c6eb415e" member_type: VOTER last_known_addr { host: "127.12.255.125" port: 46135 } } peers { permanent_uuid: "a573591e58d84e75a303105cef6c901d" member_type: VOTER last_known_addr { host: "127.12.255.124" port: 36715 } } }
08:07:03.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.411170 13845 sys_catalog.cc:458] T 00000000000000000000000000000000 P a573591e58d84e75a303105cef6c901d [sys.catalog]: This master's current role is: LEADER
08:07:03.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:03.412135 13792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:03.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:03.412775 13792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:03.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.413455 13780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: acquiring CA information for follower catalog manager: success
08:07:03.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:03.413627 13792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:03.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.432276 13780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18b2d0a700274e01940f91e1c6eb415e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:07:03.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:03.471240 13792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:03.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:03.472675 13792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.65
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.552865 13792 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:0
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:03.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.555300 13792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:03.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:03.557549 13792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:03.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:03.602351 13852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:05.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:05.498054 13715 debug-util.cc:398] Leaking SignalData structure 0x7b08000bad00 after lost signal to thread 13582
08:07:05.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:05.498739 13715 debug-util.cc:398] Leaking SignalData structure 0x7b0800082380 after lost signal to thread 13722
08:07:05.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:03.607756 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:05.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:04.981545 13850 debug-util.cc:398] Leaking SignalData structure 0x7b0800014860 after lost signal to thread 13792
08:07:05.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:05.951601 13792 thread.cc:636] GCE (cloud detector) Time spent creating pthread: real 2.346s	user 0.518s	sys 0.998s
08:07:05.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:05.953603 13792 thread.cc:602] GCE (cloud detector) Time spent starting thread: real 2.348s	user 0.518s	sys 0.999s
08:07:05.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:05.961313 13865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:05.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.962746 13792 server_base.cc:1034] running on GCE node
08:07:05.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.964069 13792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:05.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.966794 13792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:05.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.968231 13792 hybrid_clock.cc:648] HybridClock initialized: now 1786522025968048 us; error 178 us; skew 500 ppm
08:07:05.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.969563 13792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:05.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.981045 13792 webserver.cc:460] Webserver started at http://127.12.255.65:46817/ using document root <none> and password file <none>
08:07:05.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.982043 13792 fs_manager.cc:359] Metadata directory not provided
08:07:05.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.982290 13792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:05.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.982849 13792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
08:07:05.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.988065 13792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/instance:
08:07:05.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:05.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:05 on dist-test-slave-ffrd"
08:07:05.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.989360 13792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal/instance:
08:07:05.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:05.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:05 on dist-test-slave-ffrd"
08:07:05.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:05.997321 13792 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
08:07:06.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.003525 13875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:06.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.004813 13792 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.003s
08:07:06.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.005208 13792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:06.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:06.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:05 on dist-test-slave-ffrd"
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.005568 13792 fs_report.cc:383] FS layout report
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/data
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:06.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:06.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.057560 13792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:06.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.059115 13792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:06.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.059629 13792 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:06.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.062912 13792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:06.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.075125 13792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:07:06.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.075451 13792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:07:06.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.075739 13792 ts_tablet_manager.cc:610] Registered 0 tablets
08:07:06.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.075985 13792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.288038 13792 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.65:38243
08:07:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.288265 14064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.65:38243 every 8 connection(s)
08:07:06.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.291520 13792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:06.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.300570 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 13792
08:07:06.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.301255 13309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal/instance
08:07:06.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.316905 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:06.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:06.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:06.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:0
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:07:06.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:06.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:06.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:06.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:06.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:06.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.399490 14067 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:06.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.400025 14067 heartbeater.cc:464] Registering TS with master...
08:07:06.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.401108 14065 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:06.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.401340 14067 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:06.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.401455 14065 heartbeater.cc:464] Registering TS with master...
08:07:06.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.402323 14065 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:06.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.403445 14074 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:06.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.403776 14074 heartbeater.cc:464] Registering TS with master...
08:07:06.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.404529 14074 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:06.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.405189 13512 ts_manager.cc:194] Registered new tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:06.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.407406 13653 ts_manager.cc:194] Registered new tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:06.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.407537 13391 ts_manager.cc:194] Registered new tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:06.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.410883 13653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.65:54595
08:07:06.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.707026 14073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:06.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.707648 14073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:06.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.708281 14073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:06.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.742270 14073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:06.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.743263 14073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.66
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.779706 14073 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:0
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.781404 14073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:06.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.783756 14073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:06.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.809504 14104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:07.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:07.415436 14065 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:08.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.810835 14101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:08.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:06.810315 14097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:08.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:06.813021 14073 server_base.cc:1034] running on GCE node
08:07:08.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.059648 14073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:08.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.062639 14073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:08.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.064152 14073 hybrid_clock.cc:648] HybridClock initialized: now 1786522028064110 us; error 45 us; skew 500 ppm
08:07:08.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.065042 14073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:08.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.072332 14073 webserver.cc:460] Webserver started at http://127.12.255.66:42867/ using document root <none> and password file <none>
08:07:08.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.073480 14073 fs_manager.cc:359] Metadata directory not provided
08:07:08.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.073755 14073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:08.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.074293 14073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
08:07:08.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.079859 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/instance:
08:07:08.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:08.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:08 on dist-test-slave-ffrd"
08:07:08.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.081061 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal/instance:
08:07:08.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:08.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:08 on dist-test-slave-ffrd"
08:07:08.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.089267 14073 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.006s	sys 0.001s
08:07:08.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.095166 14117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:08.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.096330 14073 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.004s
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.096637 14073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:08 on dist-test-slave-ffrd"
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.096999 14073 fs_report.cc:383] FS layout report
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/data
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:08.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:08.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:08.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:08.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:08.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:08.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:08.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:08.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.146687 14073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:08.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.148417 14073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:08.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.148990 14073 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.151959 14073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.156383 14073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:07:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.156616 14073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:07:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.156991 14073 ts_tablet_manager.cc:610] Registered 0 tablets
08:07:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.157179 14073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
08:07:08.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.302906 14073 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.66:39593
08:07:08.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.303027 14306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.66:39593 every 8 connection(s)
08:07:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.305534 14073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:08.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.310428 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 14073
08:07:08.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.310948 13309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal/instance
08:07:08.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.323331 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:08.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:0
08:07:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:07:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:08.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.363976 14307 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:08.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.364490 14307 heartbeater.cc:464] Registering TS with master...
08:07:08.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.364850 14308 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:08.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.365180 14308 heartbeater.cc:464] Registering TS with master...
08:07:08.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.365960 14307 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:08.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.366063 14308 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:08.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.369194 13653 ts_manager.cc:194] Registered new tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:08.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.369997 14310 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:08.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.370337 14310 heartbeater.cc:464] Registering TS with master...
08:07:08.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.370734 13653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.66:39989
08:07:08.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.371079 14310 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:08.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.372498 13512 ts_manager.cc:194] Registered new tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.375461 13391 ts_manager.cc:194] Registered new tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:08.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.675382 14327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:08.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.676002 14327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:08.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.676625 14327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.709676 14327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.710692 14327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.67
08:07:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.747319 14327 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:0
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:08.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.748925 14327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:08.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.750726 14327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:08.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.769702 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:09.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:09.375522 14307 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:08.771319 14327 server_base.cc:1034] running on GCE node
08:07:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.769714 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:08.770304 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:09.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:09.988080 14327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:09.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:09.991164 14327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:09.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:09.992645 14327 hybrid_clock.cc:648] HybridClock initialized: now 1786522029992594 us; error 66 us; skew 500 ppm
08:07:09.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:09.993979 14327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:10.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.001266 14327 webserver.cc:460] Webserver started at http://127.12.255.67:37217/ using document root <none> and password file <none>
08:07:10.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.002625 14327 fs_manager.cc:359] Metadata directory not provided
08:07:10.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.003233 14327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:10.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.003805 14327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
08:07:10.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.009407 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/instance:
08:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:09 on dist-test-slave-ffrd"
08:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.010563 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal/instance:
08:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:10.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:09 on dist-test-slave-ffrd"
08:07:10.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.018810 14327 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.004s
08:07:10.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.024837 14359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:10.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.026187 14327 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.026618 14327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:09 on dist-test-slave-ffrd"
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.027011 14327 fs_report.cc:383] FS layout report
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/data
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:10.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:10.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.070672 14327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:10.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.072252 14327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:10.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.072705 14327 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:10.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.076045 14327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:10.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.080814 14327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:07:10.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.081106 14327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
08:07:10.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.081452 14327 ts_tablet_manager.cc:610] Registered 0 tablets
08:07:10.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.081666 14327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:07:10.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.233987 14327 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.67:43597
08:07:10.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.234093 14551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.67:43597 every 8 connection(s)
08:07:10.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.236474 14327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:10.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.243294 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 14327
08:07:10.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.243790 13309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal/instance
08:07:10.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.292465 14553 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:10.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.292984 14553 heartbeater.cc:464] Registering TS with master...
08:07:10.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.294198 14553 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:10.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.297304 13512 ts_manager.cc:194] Registered new tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.301685 14552 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.301736 14554 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.302163 14552 heartbeater.cc:464] Registering TS with master...
08:07:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.302161 14554 heartbeater.cc:464] Registering TS with master...
08:07:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.303442 14554 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:10.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.303426 14552 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:10.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.306123 13653 ts_manager.cc:194] Registered new tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:10.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.308570 13653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.67:56049
08:07:10.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.311036 13391 ts_manager.cc:194] Registered new tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:10.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:10.315415 13309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
08:07:10.326 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:07:10.328 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

08:07:10.338 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.12.255.126"
      port: 39387
    }
    bound_http_address {
      host: "127.12.255.126"
      port: 45131
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.12.255.125"
      port: 46135
    }
    bound_http_address {
      host: "127.12.255.125"
      port: 37307
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.12.255.124"
      port: 36715
    }
    bound_http_address {
      host: "127.12.255.124"
      port: 45627
    }
  }
}

08:07:10.346 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

08:07:10.350 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.12.255.65"
      port: 38243
    }
    bound_http_address {
      host: "127.12.255.65"
      port: 46817
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.12.255.66"
      port: 39593
    }
    bound_http_address {
      host: "127.12.255.66"
      port: 42867
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.12.255.67"
      port: 43597
    }
    bound_http_address {
      host: "127.12.255.67"
      port: 37217
    }
  }
}

08:07:10.350 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
08:07:10.549 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
08:07:10.770 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.12.255.125:46135(127.12.255.125:46135)] connecting to peer
08:07:10.833 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.12.255.124:36715(127.12.255.124:36715)] connecting to peer
08:07:10.835 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.12.255.126:39387(127.12.255.126:39387)] connecting to peer
08:07:10.881 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.12.255.124:36715(127.12.255.124:36715)] Successfully connected to peer
08:07:10.885 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.12.255.125:46135(127.12.255.125:46135)] Successfully connected to peer
08:07:10.959 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.12.255.126:39387(127.12.255.126:39387)] Successfully connected to peer
08:07:10.988 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.12.255.125
08:07:10.989 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.12.255.124
08:07:11.110 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.12.255.126
08:07:11.246 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x84ca06db, L:/127.0.0.1:42350 - R:/127.12.255.124:36715] using SASL/PLAIN
08:07:11.246 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x1bdbf97a, L:/127.0.0.1:38556 - R:/127.12.255.125:46135] using SASL/PLAIN
08:07:11.249 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x183ac42c, L:/127.0.0.1:38142 - R:/127.12.255.126:39387] using SASL/PLAIN
08:07:11.293 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
08:07:11.294 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=583), Traces: [0ms] refreshing cache from master, [41ms] Sub RPC ConnectToMaster: sending RPC to server master-127.12.255.125:46135, [116ms] Sub RPC ConnectToMaster: sending RPC to server master-127.12.255.124:36715, [117ms] Sub RPC ConnectToMaster: sending RPC to server master-127.12.255.126:39387, [546ms] Sub RPC ConnectToMaster: received response from server master-127.12.255.125:46135: OK, [546ms] Sub RPC ConnectToMaster: received response from server master-127.12.255.124:36715: OK, [548ms] Sub RPC ConnectToMaster: received response from server master-127.12.255.126:39387: OK, deferred=null) after lookup
08:07:11.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.316344 14552 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.351257 13653 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42350:
08:07:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695"
08:07:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
08:07:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:07:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
08:07:11.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:07:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:07:11.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:07:11.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:11.355154 13653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
08:07:11.418 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 53bc6cdae2994b4dbc03d481215a9b8c
08:07:11.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.436790 14441 tablet_service.cc:1467] Processing CreateTablet for tablet ccffce57a953436db65493227b7dd42b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695 [id=53bc6cdae2994b4dbc03d481215a9b8c]), partition=RANGE (key) PARTITION UNBOUNDED
08:07:11.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.438753 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccffce57a953436db65493227b7dd42b. 1 dirs total, 0 dirs full, 0 dirs failed
08:07:11.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.451072 13960 tablet_service.cc:1467] Processing CreateTablet for tablet ccffce57a953436db65493227b7dd42b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695 [id=53bc6cdae2994b4dbc03d481215a9b8c]), partition=RANGE (key) PARTITION UNBOUNDED
08:07:11.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.451074 14192 tablet_service.cc:1467] Processing CreateTablet for tablet ccffce57a953436db65493227b7dd42b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695 [id=53bc6cdae2994b4dbc03d481215a9b8c]), partition=RANGE (key) PARTITION UNBOUNDED
08:07:11.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.452838 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccffce57a953436db65493227b7dd42b. 1 dirs total, 0 dirs full, 0 dirs failed
08:07:11.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.452884 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccffce57a953436db65493227b7dd42b. 1 dirs total, 0 dirs full, 0 dirs failed
08:07:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.476492 14605 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap starting.
08:07:11.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.479234 14606 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap starting.
08:07:11.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.483809 14605 tablet_bootstrap.cc:654] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Neither blocks nor log segments found. Creating new log.
08:07:11.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.485514 14609 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap starting.
08:07:11.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.486217 14605 log.cc:824] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Log is configured to *not* fsync() on all Append() calls
08:07:11.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.486397 14606 tablet_bootstrap.cc:654] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Neither blocks nor log segments found. Creating new log.
08:07:11.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.489192 14606 log.cc:824] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Log is configured to *not* fsync() on all Append() calls
08:07:11.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.492439 14605 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: No bootstrap required, opened a new log
08:07:11.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.492973 14605 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.010s
08:07:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.496296 14609 tablet_bootstrap.cc:654] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Neither blocks nor log segments found. Creating new log.
08:07:11.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.498435 14609 log.cc:824] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Log is configured to *not* fsync() on all Append() calls
08:07:11.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.499338 14606 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: No bootstrap required, opened a new log
08:07:11.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.499856 14606 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.004s
08:07:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.503824 14609 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: No bootstrap required, opened a new log
08:07:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.504252 14609 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.002s
08:07:11.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.523598 14605 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.524410 14605 raft_consensus.cc:374] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:07:11.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.524785 14605 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e631edcb842e4e809e7bc9f4a63a388c, State: Initialized, Role: FOLLOWER
08:07:11.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.525758 14605 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.530124 14605 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent starting tablet: real 0.036s	user 0.031s	sys 0.003s
08:07:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.531299 14609 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.532475 14609 raft_consensus.cc:374] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:07:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.532773 14609 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Initialized, Role: FOLLOWER
08:07:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.533797 14609 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.534665 14606 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.535679 14606 raft_consensus.cc:374] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:07:11.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.536068 14606 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Initialized, Role: FOLLOWER
08:07:11.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.537106 14606 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.555441 14609 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent starting tablet: real 0.051s	user 0.031s	sys 0.004s
08:07:11.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.556155 14606 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent starting tablet: real 0.056s	user 0.044s	sys 0.008s
08:07:11.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:11.595410 14325 tablet.cc:2369] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:07:11.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:11.652168 14078 tablet.cc:2369] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:07:11.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.689049 14625 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:11.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.689653 14625 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.692181 14625 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:11.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.706813 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:11.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.706858 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c" is_pre_election: true
08:07:11.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.707620 14231 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 0.
08:07:11.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.707647 14474 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 0.
08:07:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.708802 13877 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: 
08:07:11.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.709630 14625 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
08:07:11.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.709897 14625 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:07:11.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.710111 14625 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 0 FOLLOWER]: Advancing to term 1
08:07:11.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.714931 14625 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.716449 14625 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 1 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:11.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.717417 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:11.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.717552 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:11.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.717993 14474 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 0 FOLLOWER]: Advancing to term 1
08:07:11.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.718078 14231 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 0 FOLLOWER]: Advancing to term 1
08:07:11.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.724782 14474 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 1.
08:07:11.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.724782 14231 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 1.
08:07:11.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.725878 13877 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: 
08:07:11.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.726562 14625 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 1 FOLLOWER]: Leader election won for term 1
08:07:11.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.728268 14625 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 1 LEADER]: Becoming Leader. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Running, Role: LEADER
08:07:11.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.729259 14625 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:11.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:11.741463 13653 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65). New cstate: current_term: 1 leader_uuid: "ec3940bcabac4a23852b5f27573cd9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: HEALTHY } } }
08:07:11.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:11.770664 14560 tablet.cc:2369] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:07:11.881 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 53bc6cdae2994b4dbc03d481215a9b8c
08:07:11.908 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:11.915 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299992, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:11.918 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [11ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=null) after lookup
08:07:11.934 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:11.935 [DEBUG - kudu-nio-1] (Connection.java:757) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] Successfully connected to peer
08:07:12.001 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x473475e8, L:/127.0.0.1:35586 - R:/127.12.255.65:38243] using TOKEN/null
08:07:12.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:12.019807 14231 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Refusing update from remote peer ec3940bcabac4a23852b5f27573cd9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:07:12.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:12.019809 14474 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Refusing update from remote peer ec3940bcabac4a23852b5f27573cd9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:07:12.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:12.021456 14628 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:07:12.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:12.022078 14625 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:07:12.208 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:12.210 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.65:38243
08:07:12.211 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:12.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:12.212116 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 13792
08:07:12.267 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:12.267 [DEBUG - kudu-nio-1] (Connection.java:239) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] handling channelInactive
08:07:12.268 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state READY due to: connection closed
08:07:12.269 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: connection closed
08:07:12.278 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:12.278 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:12.279 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@349232021(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@487cea64 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1eb067f7 -> wakeup thread Time-limited test)) after lookup
08:07:12.280 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:12.282 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.282 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.282 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.297 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:12.297 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:12.298 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [12ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [15ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [16ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@349232021(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@487cea64 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1eb067f7 -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630))) after lookup
08:07:12.299 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:12.300 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.300 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.300 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.337 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:12.337 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:12.338 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [12ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [15ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [16ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [32ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [33ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [34ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@349232021(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@487cea64 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1eb067f7 -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655))) after lookup
08:07:12.339 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:12.341 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.341 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.341 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.457 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:12.457 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:12.458 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [12ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [15ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [16ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [32ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [33ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [34ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [72ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [74ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [74ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [189ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@349232021(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@487cea64 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1eb067f7 -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309))) after lookup
08:07:12.459 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:12.460 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.460 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.460 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.537 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:12.537 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:12.538 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=270), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [12ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [15ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [16ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [32ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [33ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [34ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [72ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [74ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [74ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [189ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [192ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [193ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [193ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [265ms] refreshing cache from master, [265ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [269ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@349232021(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@487cea64 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309) -> (continuation of Deferred@577721202 after retry RPC@1590024823), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1eb067f7 -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309) -> (continuation of Deferred@577721202 after retry RPC@1590024823))) after lookup
08:07:12.539 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:12.541 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.541 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:12.542 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:13.097 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:13.097 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:13.098 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=830), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [12ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [15ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [16ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [32ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [33ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [34ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [72ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [74ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [74ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [189ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [192ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [193ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [193ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [265ms] refreshing cache from master, [265ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [269ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [272ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [275ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [275ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [825ms] refreshing cache from master, [825ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [829ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@349232021(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@487cea64 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309) -> (continuation of Deferred@577721202 after retry RPC@1590024823) -> (continuation of Deferred@621820803 after retry RPC@1433815860), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1eb067f7 -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309) -> (continuation of Deferred@577721202 after retry RPC@1590024823) -> (continuation of Deferred@621820803 after retry RPC@1433815860))) after lookup
08:07:13.099 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:13.100 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:13.100 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:13.100 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:13.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.651955 14647 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:13.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.652494 14647 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:13.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.653941 14649 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:13.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.654378 14649 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:13.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.655539 14647 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:13.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.656971 14649 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:13.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:13.671530 14364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:13.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:13.672639 14123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:13.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:13.680413 14123 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:13.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:13.683583 14364 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:13.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.686081 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:13.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.686278 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c" is_pre_election: true
08:07:13.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.686553 14474 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e631edcb842e4e809e7bc9f4a63a388c in term 1.
08:07:13.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.686884 14231 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 1.
08:07:13.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.687775 14121 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:13.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.688134 14362 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:13.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.688760 14647 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Leader pre-election won for term 2
08:07:13.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.689100 14647 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Starting leader election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:13.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.689261 14649 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Leader pre-election won for term 2
08:07:13.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.689308 14647 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 1 FOLLOWER]: Advancing to term 2
08:07:13.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.689486 14649 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:13.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.689700 14649 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 1 FOLLOWER]: Advancing to term 2
08:07:13.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.693646 14647 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:13.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.695168 14647 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 2 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:13.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.694821 14649 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:13.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.696064 14474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:13.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.696820 14474 raft_consensus.cc:2379] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e631edcb842e4e809e7bc9f4a63a388c in current term 2: Already voted for candidate ff4960fadb0549a59a1e29815dde2bb0 in this term.
08:07:13.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.697693 14649 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 2 election: Requested vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:13.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.699122 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:13.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:13.699616 14123 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:13.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.699888 14231 raft_consensus.cc:2379] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ff4960fadb0549a59a1e29815dde2bb0 in current term 2: Already voted for candidate e631edcb842e4e809e7bc9f4a63a388c in this term.
08:07:13.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.700026 14123 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c; no voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0
08:07:13.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.700774 14647 raft_consensus.cc:2726] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
08:07:13.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:13.701506 14364 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:13.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.701792 14364 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1
08:07:13.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:13.702354 14649 raft_consensus.cc:2726] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
08:07:15.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.301842 14665 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:15.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.302649 14665 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:15.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.304757 14665 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:15.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.305575 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c" is_pre_election: true
08:07:15.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.306213 14231 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 2.
08:07:15.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.307564 14362 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: 
08:07:15.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.310689 14665 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Leader pre-election won for term 3
08:07:15.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.311100 14665 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:07:15.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.311482 14665 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 2 FOLLOWER]: Advancing to term 3
08:07:15.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:15.315924 14364 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:15.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.317684 14665 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:15.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.319192 14665 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 3 election: Requested vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.320066 14231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.320639 14231 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 2 FOLLOWER]: Advancing to term 3
08:07:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:15.323232 14364 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.325695 14231 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 3.
08:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.326508 14362 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.327209 14665 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Leader election won for term 3
08:07:15.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.328111 14665 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 LEADER]: Becoming Leader. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Running, Role: LEADER
08:07:15.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.329011 14665 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:15.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.338245 13653 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 reported cstate change: term changed from 1 to 3, leader changed from ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65) to ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67). New cstate: current_term: 3 leader_uuid: "ff4960fadb0549a59a1e29815dde2bb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: UNKNOWN } } }
08:07:15.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:15.829139 14364 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 -> Peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Couldn't send request to peer ec3940bcabac4a23852b5f27573cd9a1. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:15.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:15.866524 14545 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5520 after lost signal to thread 14333
08:07:15.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:15.867479 14545 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce780 after lost signal to thread 14551
08:07:15.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.891561 14231 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 3 FOLLOWER]: Refusing update from remote peer ff4960fadb0549a59a1e29815dde2bb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
08:07:15.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:15.893133 14665 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
08:07:16.777 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:16.777 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:16.781 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4510), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [12ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [15ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [16ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [32ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [33ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [34ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [72ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [74ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [74ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [189ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [192ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [193ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [193ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [265ms] refreshing cache from master, [265ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [269ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [272ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [275ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [275ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [825ms] refreshing cache from master, [825ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [829ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [831ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [833ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [833ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4505ms] refreshing cache from master, [4505ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [4509ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@349232021(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@487cea64 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309) -> (continuation of Deferred@577721202 after retry RPC@1590024823) -> (continuation of Deferred@621820803 after retry RPC@1433815860) -> (continuation of Deferred@1541275094 after retry RPC@1617127587), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1eb067f7 -> wakeup thread Time-limited test -> (continuation of Deferred@1254893495 after retry RPC@1465810630) -> (continuation of Deferred@342076108 after retry RPC@1830505655) -> (continuation of Deferred@658626094 after retry RPC@268978309) -> (continuation of Deferred@577721202 after retry RPC@1590024823) -> (continuation of Deferred@621820803 after retry RPC@1433815860) -> (continuation of Deferred@1541275094 after retry RPC@1617127587))) after lookup
08:07:16.782 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:16.794 [DEBUG - kudu-nio-0] (Connection.java:757) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] Successfully connected to peer
08:07:16.826 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xd285802a, L:/127.0.0.1:57672 - R:/127.12.255.67:43597] using TOKEN/null
08:07:16.891 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.65:38243
08:07:16.893 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:16.895740 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.255173 14722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.255653 14722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:17.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.256083 14722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:17.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.286888 14722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:17.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.288018 14722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.65
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:17.343575 14722 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:17.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:17.345417 14722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:17.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:17.347491 14722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.375083 14733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:18.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:18.138617 14364 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 -> Peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Couldn't send request to peer ec3940bcabac4a23852b5f27573cd9a1. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:07:18.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.378036 14739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:18.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:17.375946 14732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:18.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:18.777925 14737 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1401 milliseconds
08:07:18.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.778039 14722 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:07:18.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.779342 14722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:18.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.781888 14722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:18.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.783289 14722 hybrid_clock.cc:648] HybridClock initialized: now 1786522038783255 us; error 50 us; skew 500 ppm
08:07:18.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.784174 14722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:18.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.796743 14722 webserver.cc:460] Webserver started at http://127.12.255.65:46817/ using document root <none> and password file <none>
08:07:18.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.797823 14722 fs_manager.cc:359] Metadata directory not provided
08:07:18.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.798103 14722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:18.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.806967 14722 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.004s	sys 0.000s
08:07:18.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.811791 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:18.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.813091 14722 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:07:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.813413 14722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:05 on dist-test-slave-ffrd"
08:07:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.816151 14722 fs_report.cc:383] FS layout report
08:07:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:18.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/data
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:18.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:18.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.862776 14722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:18.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.864795 14722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:18.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.865371 14722 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:18.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.868744 14722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:18.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.876336 14773 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:18.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.884589 14722 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:18.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.884819 14722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
08:07:18.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.885135 14722 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:18.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.890309 14722 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:18.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.890534 14722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.004s	sys 0.000s
08:07:18.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.890827 14773 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap starting.
08:07:18.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:18.973111 14773 log.cc:824] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Log is configured to *not* fsync() on all Append() calls
08:07:19.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.002535 14773 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:19.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.003763 14773 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap complete.
08:07:19.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.005867 14773 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent bootstrapping tablet: real 0.115s	user 0.074s	sys 0.023s
08:07:19.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.030727 14773 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.033711 14773 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Initialized, Role: FOLLOWER
08:07:19.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.034960 14773 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:19.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.039482 14773 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent starting tablet: real 0.033s	user 0.018s	sys 0.015s
08:07:19.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.095564 14722 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.65:38243
08:07:19.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.095830 14962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.65:38243 every 8 connection(s)
08:07:19.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.098420 14722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:19.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.103039 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 14722
08:07:19.104 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:19.146 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:19.149 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.67:43597
08:07:19.149 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:07:19.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.150166 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 14327
08:07:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.193593 14965 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.194136 14965 heartbeater.cc:464] Registering TS with master...
08:07:19.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.195459 14965 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:19.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.196499 14963 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:19.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.196854 14963 heartbeater.cc:464] Registering TS with master...
08:07:19.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.195369 14964 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:19.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.197463 14964 heartbeater.cc:464] Registering TS with master...
08:07:19.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.197978 14963 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:19.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.198273 14964 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:19.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.201440 13653 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:19.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.202009 13390 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:19.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.202507 13512 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:19.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:19.204687 13653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.65:39715
08:07:19.231 [DEBUG - kudu-nio-0] (Connection.java:239) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] handling channelInactive
08:07:19.231 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state READY due to: connection closed
08:07:19.232 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:19.233 [DEBUG - Time-limited test] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:19.234 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.235 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.235 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.257 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:19.257 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:19.258 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1941409962(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@7d84fcb8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5793f8d2 -> wakeup thread Time-limited test)) after lookup
08:07:19.259 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:19.260 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.261 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.261 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.276 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:19.277 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:19.277 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [29ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1941409962(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@7d84fcb8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5793f8d2 -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641))) after lookup
08:07:19.279 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:19.280 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.280 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.280 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.318 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:19.319 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:19.320 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=87), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [29ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1941409962(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@7d84fcb8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5793f8d2 -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948))) after lookup
08:07:19.320 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:19.322 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.322 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.322 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.417 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:19.417 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:19.418 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=185), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [29ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [88ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [90ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [91ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [180ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [184ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1941409962(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@7d84fcb8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948) -> (continuation of Deferred@1741679870 after retry RPC@1924916358), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5793f8d2 -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948) -> (continuation of Deferred@1741679870 after retry RPC@1924916358))) after lookup
08:07:19.419 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:19.420 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.420 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.421 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.738 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:19.738 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:19.739 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=506), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [29ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [88ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [90ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [91ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [180ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [184ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [187ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [188ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [189ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [500ms] refreshing cache from master, [501ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [505ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1941409962(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@7d84fcb8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948) -> (continuation of Deferred@1741679870 after retry RPC@1924916358) -> (continuation of Deferred@1186294158 after retry RPC@2095325933), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5793f8d2 -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948) -> (continuation of Deferred@1741679870 after retry RPC@1924916358) -> (continuation of Deferred@1186294158 after retry RPC@2095325933))) after lookup
08:07:19.740 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:19.742 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.742 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:19.742 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:20.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.209163 14963 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:20.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.411769 14983 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff4960fadb0549a59a1e29815dde2bb0)
08:07:20.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.412371 14983 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:20.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.414498 14983 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:20.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:20.417586 14121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111) [suppressed 1 similar messages]
08:07:20.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:20.421479 14121 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111)
08:07:20.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.428257 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1" is_pre_election: true
08:07:20.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.429024 14875 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e631edcb842e4e809e7bc9f4a63a388c in term 1.
08:07:20.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.430197 14123 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: ff4960fadb0549a59a1e29815dde2bb0
08:07:20.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.430836 14983 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 3 FOLLOWER]: Leader pre-election won for term 4
08:07:20.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.431152 14983 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff4960fadb0549a59a1e29815dde2bb0)
08:07:20.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.431520 14983 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 3 FOLLOWER]: Advancing to term 4
08:07:20.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.437089 14983 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:20.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.438709 14983 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 4 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:20.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.439934 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:20.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.440573 14875 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 1 FOLLOWER]: Advancing to term 4
08:07:20.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:20.442019 14121 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111)
08:07:20.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.447804 14875 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e631edcb842e4e809e7bc9f4a63a388c in term 4.
08:07:20.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.448917 14123 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: ff4960fadb0549a59a1e29815dde2bb0
08:07:20.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.449683 14983 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Leader election won for term 4
08:07:20.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.450909 14983 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 LEADER]: Becoming Leader. State: Replica: e631edcb842e4e809e7bc9f4a63a388c, State: Running, Role: LEADER
08:07:20.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.452476 14983 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:20.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.462965 13653 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c reported cstate change: term changed from 3 to 4, leader changed from ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67) to e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66). New cstate: current_term: 4 leader_uuid: "e631edcb842e4e809e7bc9f4a63a388c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: UNKNOWN } } }
08:07:20.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:20.836879 14121 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c -> Peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Couldn't send request to peer ff4960fadb0549a59a1e29815dde2bb0. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:20.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.920725 14875 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 4 FOLLOWER]: Refusing update from remote peer e631edcb842e4e809e7bc9f4a63a388c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
08:07:20.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:20.922310 14983 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
08:07:21.538 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:21.538 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:21.539 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2306), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [29ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [88ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [90ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [91ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [180ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [184ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [187ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [188ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [189ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [500ms] refreshing cache from master, [501ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [505ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [508ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [510ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [511ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [2300ms] refreshing cache from master, [2300ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [2305ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1941409962(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@7d84fcb8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948) -> (continuation of Deferred@1741679870 after retry RPC@1924916358) -> (continuation of Deferred@1186294158 after retry RPC@2095325933) -> (continuation of Deferred@1101354752 after retry RPC@478930579), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5793f8d2 -> wakeup thread Time-limited test -> (continuation of Deferred@1973682975 after retry RPC@1035546641) -> (continuation of Deferred@1099640917 after retry RPC@1733182948) -> (continuation of Deferred@1741679870 after retry RPC@1924916358) -> (continuation of Deferred@1186294158 after retry RPC@2095325933) -> (continuation of Deferred@1101354752 after retry RPC@478930579))) after lookup
08:07:21.540 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:21.541 [DEBUG - kudu-nio-1] (Connection.java:757) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] Successfully connected to peer
08:07:21.561 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x403935cf, L:/127.0.0.1:50930 - R:/127.12.255.66:39593] using TOKEN/null
08:07:21.611 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.67:43597
08:07:21.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:21.613673 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:43597
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37217
08:07:21.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:21.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:21.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:21.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:21.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:21.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:21.945068 15040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:21.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:21.945566 15040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:21.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:21.946019 15040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:21.983565 15040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:21.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:21.984409 15040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.67
08:07:22.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:22.041456 15040 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:43597
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37217
08:07:22.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:22.043084 15040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:22.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:22.045020 15040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:22.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:22.069365 15051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:23.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:22.073069 15040 server_base.cc:1034] running on GCE node
08:07:23.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:22.075397 15057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:23.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:22.069653 15055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:23.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.306290 15040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:23.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:23.311080 14121 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c -> Peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Couldn't send request to peer ff4960fadb0549a59a1e29815dde2bb0. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:07:23.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.313539 15040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:23.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.315011 15040 hybrid_clock.cc:648] HybridClock initialized: now 1786522043314863 us; error 142 us; skew 500 ppm
08:07:23.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.316829 15040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:23.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.324934 15040 webserver.cc:460] Webserver started at http://127.12.255.67:37217/ using document root <none> and password file <none>
08:07:23.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.325947 15040 fs_manager.cc:359] Metadata directory not provided
08:07:23.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.326237 15040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.334595 15040 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
08:07:23.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.339290 15079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:23.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.340481 15040 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.340773 15040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:09 on dist-test-slave-ffrd"
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.343168 15040 fs_report.cc:383] FS layout report
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/data
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:23.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:23.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.409348 15040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:23.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.410929 15040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:23.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.411368 15040 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:23.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.414224 15040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.421123 15089 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:23.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.429394 15040 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:23.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.429647 15040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
08:07:23.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.429891 15040 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:23.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.435212 15040 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:23.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.435459 15040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
08:07:23.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.435988 15089 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap starting.
08:07:23.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.501482 15089 log.cc:824] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Log is configured to *not* fsync() on all Append() calls
08:07:23.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.543416 15089 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:23.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.544353 15089 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap complete.
08:07:23.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.545980 15089 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent bootstrapping tablet: real 0.110s	user 0.073s	sys 0.032s
08:07:23.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.566421 15089 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:23.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.569487 15089 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Initialized, Role: FOLLOWER
08:07:23.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.570554 15089 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:23.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.575387 15089 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent starting tablet: real 0.029s	user 0.018s	sys 0.009s
08:07:23.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.617823 15040 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.67:43597
08:07:23.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.618080 15285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.67:43597 every 8 connection(s)
08:07:23.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.620589 15040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:23.624 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:23.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.623238 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 15040
08:07:23.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.674342 15288 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:23.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.674882 15288 heartbeater.cc:464] Registering TS with master...
08:07:23.675 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:23.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.676133 15288 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:23.678 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.66:39593
08:07:23.678 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:07:23.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.679196 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 14073
08:07:23.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.680114 13653 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:23.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.684017 13653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.67:55403
08:07:23.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.693709 15289 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:23.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.694152 15289 heartbeater.cc:464] Registering TS with master...
08:07:23.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.695082 15289 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:23.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.696666 15292 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:23.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.697049 15292 heartbeater.cc:464] Registering TS with master...
08:07:23.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.697763 15292 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:23.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.699007 13511 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:23.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:23.701398 13391 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:23.725 [DEBUG - kudu-nio-1] (Connection.java:239) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] handling channelInactive
08:07:23.726 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state READY due to: connection closed
08:07:23.726 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:23.727 [DEBUG - Time-limited test] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:23.729 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.729 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.729 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.737 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:23.737 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:23.737 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=10), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1512972028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@221e57b0 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fc4387a -> wakeup thread Time-limited test)) after lookup
08:07:23.738 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:23.739 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.739 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.739 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.756 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:23.756 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:23.757 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [11ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [12ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [12ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1512972028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@221e57b0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fc4387a -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545))) after lookup
08:07:23.758 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:23.759 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.760 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.760 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.796 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:23.796 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:23.797 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [11ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [12ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [12ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [31ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [33ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [33ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1512972028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@221e57b0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fc4387a -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770))) after lookup
08:07:23.797 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:23.798 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.798 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.798 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.876 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:23.876 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:23.877 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=149), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [11ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [12ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [12ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [31ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [33ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [33ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [70ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [71ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [71ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [148ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1512972028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@221e57b0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770) -> (continuation of Deferred@1798409614 after retry RPC@1254904177), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fc4387a -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770) -> (continuation of Deferred@1798409614 after retry RPC@1254904177))) after lookup
08:07:23.878 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:23.880 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.880 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:23.880 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:24.337 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:24.337 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:24.338 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=611), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [11ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [12ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [12ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [31ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [33ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [33ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [70ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [71ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [71ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [148ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [150ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [153ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [153ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [605ms] refreshing cache from master, [605ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [609ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1512972028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@221e57b0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770) -> (continuation of Deferred@1798409614 after retry RPC@1254904177) -> (continuation of Deferred@1420064637 after retry RPC@2085348161), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fc4387a -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770) -> (continuation of Deferred@1798409614 after retry RPC@1254904177) -> (continuation of Deferred@1420064637 after retry RPC@2085348161))) after lookup
08:07:24.338 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:24.339 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:24.339 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:24.340 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:24.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:24.693275 15288 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:24.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:24.837921 15305 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:24.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:24.838449 15305 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:24.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:24.840734 15305 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:24.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:24.847214 15082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:24.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:24.851874 15082 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:24.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:24.858125 14875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1" is_pre_election: true
08:07:24.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:24.860176 15084 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1
08:07:24.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:24.861100 15305 raft_consensus.cc:2726] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
08:07:25.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.128312 15313 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e631edcb842e4e809e7bc9f4a63a388c)
08:07:25.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.128747 15313 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:25.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.131057 15313 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:25.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:25.135499 14761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:25.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:25.140761 14761 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:25.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.144455 15213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:25.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.145419 15213 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 3.
08:07:25.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.146914 14762 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c
08:07:25.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.147704 15313 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 4 FOLLOWER]: Leader pre-election won for term 5
08:07:25.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.148062 15313 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e631edcb842e4e809e7bc9f4a63a388c)
08:07:25.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.148397 15313 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 4 FOLLOWER]: Advancing to term 5
08:07:25.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.153134 15313 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:25.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.154819 15313 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 5 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:25.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.155872 15213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:25.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.156409 15213 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 3 FOLLOWER]: Advancing to term 5
08:07:25.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:25.158808 14761 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:25.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.164289 15213 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 5.
08:07:25.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.165261 14762 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c
08:07:25.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.165964 15313 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 5 FOLLOWER]: Leader election won for term 5
08:07:25.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.168280 15313 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 5 LEADER]: Becoming Leader. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Running, Role: LEADER
08:07:25.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.169749 15313 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:25.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.178668 13653 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 reported cstate change: term changed from 4 to 5, leader changed from e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66) to ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65). New cstate: current_term: 5 leader_uuid: "ec3940bcabac4a23852b5f27573cd9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: HEALTHY } } }
08:07:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:25.616215 14761 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 -> Peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Couldn't send request to peer e631edcb842e4e809e7bc9f4a63a388c. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:25.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.625324 15213 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 5 FOLLOWER]: Refusing update from remote peer ec3940bcabac4a23852b5f27573cd9a1: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
08:07:25.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.626744 15313 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
08:07:25.697 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:25.697 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:25.698 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1970), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [9ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [11ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [12ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [12ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [29ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [31ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [33ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [33ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [69ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [70ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [71ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [71ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [148ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [150ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [153ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [153ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [605ms] refreshing cache from master, [605ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [609ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [611ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [613ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [613ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [1965ms] refreshing cache from master, [1965ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1969ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1512972028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@221e57b0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770) -> (continuation of Deferred@1798409614 after retry RPC@1254904177) -> (continuation of Deferred@1420064637 after retry RPC@2085348161) -> (continuation of Deferred@1714210200 after retry RPC@1094886932), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fc4387a -> wakeup thread Time-limited test -> (continuation of Deferred@1533125974 after retry RPC@816423545) -> (continuation of Deferred@2011027554 after retry RPC@169477770) -> (continuation of Deferred@1798409614 after retry RPC@1254904177) -> (continuation of Deferred@1420064637 after retry RPC@2085348161) -> (continuation of Deferred@1714210200 after retry RPC@1094886932))) after lookup
08:07:25.699 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:25.700 [DEBUG - kudu-nio-0] (Connection.java:757) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] Successfully connected to peer
08:07:25.721 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x20208375, L:/127.0.0.1:41872 - R:/127.12.255.65:38243] using TOKEN/null
08:07:25.765 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.66:39593
08:07:25.765 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:25.767058 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:39593
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:25.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:25.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42867
08:07:25.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:25.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:25.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:25.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:25.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:26.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.170212 15360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:26.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.170703 15360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:26.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.171142 15360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:26.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.205884 15360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:26.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.207039 15360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.66
08:07:26.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:26.244240 15360 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:26.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:39593
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42867
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:26.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:26.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:26.245817 15360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:26.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:26.247663 15360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:26.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.270591 15373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:27.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.270707 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:27.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:26.271118 15374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:27.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:26.272490 15360 server_base.cc:1034] running on GCE node
08:07:27.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.505349 15360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:27.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.508277 15360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:27.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.509723 15360 hybrid_clock.cc:648] HybridClock initialized: now 1786522047509693 us; error 45 us; skew 500 ppm
08:07:27.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.510519 15360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:27.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.522512 15360 webserver.cc:460] Webserver started at http://127.12.255.66:42867/ using document root <none> and password file <none>
08:07:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.523465 15360 fs_manager.cc:359] Metadata directory not provided
08:07:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.523696 15360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:27.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.531764 15360 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
08:07:27.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.536686 15394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:27.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.537937 15360 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:07:27.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.538314 15360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:27.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:27.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:08 on dist-test-slave-ffrd"
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.540729 15360 fs_report.cc:383] FS layout report
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/data
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:27.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:27.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.575526 15360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:27.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.577327 15360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:27.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.577898 15360 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:27.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.580691 15360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:27.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.586634 15408 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:27.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.594933 15360 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:27.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.595148 15360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.000s
08:07:27.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.595387 15360 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:27.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.600257 15360 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:27.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.600441 15360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
08:07:27.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.600924 15408 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap starting.
08:07:27.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.671607 15408 log.cc:824] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Log is configured to *not* fsync() on all Append() calls
08:07:27.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.752441 15408 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:27.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.753901 15408 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap complete.
08:07:27.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.756551 15408 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent bootstrapping tablet: real 0.156s	user 0.116s	sys 0.032s
08:07:27.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.791584 15408 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:27.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.797178 15408 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: e631edcb842e4e809e7bc9f4a63a388c, State: Initialized, Role: FOLLOWER
08:07:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.798601 15408 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:27.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.804745 15408 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent starting tablet: real 0.048s	user 0.033s	sys 0.011s
08:07:27.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.808674 15360 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.66:39593
08:07:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.808909 15576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.66:39593 every 8 connection(s)
08:07:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.811510 15360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:27.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.814400 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 15360
08:07:27.815 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:27.848 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:27.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.862042 15577 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:27.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.862649 15577 heartbeater.cc:464] Registering TS with master...
08:07:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.863898 15577 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:27.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.868259 13651 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:27.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.871878 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.66:56441
08:07:27.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.879223 15579 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:27.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.879640 15579 heartbeater.cc:464] Registering TS with master...
08:07:27.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.880546 15579 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:27.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.885038 13511 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:27.901 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.65:38243
08:07:27.901 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:27.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.902154 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 14722
08:07:27.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.913671 15580 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:27.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.914098 15580 heartbeater.cc:464] Registering TS with master...
08:07:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.914848 15580 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:27.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:27.918294 13391 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:27.949 [DEBUG - kudu-nio-0] (Connection.java:239) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] handling channelInactive
08:07:27.950 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state READY due to: connection closed
08:07:27.951 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:27.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:27.951188 15591 negotiation.cc:337] Failed RPC negotiation. Trace:
08:07:27.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:27.886327 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.255.65:40597 (local address 127.12.255.66:39593)
08:07:27.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:27.911092 (+ 24765us) server_negotiation.cc:206] Beginning negotiation
08:07:27.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:27.911123 (+    31us) server_negotiation.cc:399] Waiting for connection header
08:07:27.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:27.949579 (+ 38456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.255.65:40597: BlockingRecv error: recv got EOF from 127.12.255.65:40597 (error 108)
08:07:27.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":24161,"thread_start_us":24127,"threads_started":1}
08:07:27.952 [DEBUG - Time-limited test] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:27.954 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.954 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.954 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.977 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:27.977 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:27.977 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [3ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1480674920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23b17109 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@581a19f6 -> wakeup thread Time-limited test)) after lookup
08:07:27.978 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:27.980 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.980 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.980 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.996 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:27.997 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:27.997 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [3ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [27ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [29ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [29ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1480674920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23b17109 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@581a19f6 -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093))) after lookup
08:07:27.997 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:27.998 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.998 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:27.998 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.037 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:28.037 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:28.038 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=86), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [3ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [27ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [29ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [29ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [47ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [86ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1480674920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23b17109 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@581a19f6 -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315))) after lookup
08:07:28.039 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:28.041 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.041 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.041 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.135 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:28.136 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:28.136 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=185), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [3ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [27ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [29ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [29ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [47ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [86ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [87ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [90ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [90ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [180ms] refreshing cache from master, [181ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [184ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1480674920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23b17109 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@581a19f6 -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642))) after lookup
08:07:28.137 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:28.138 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.138 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.138 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.618 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:28.618 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:28.619 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=667), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [3ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [27ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [29ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [29ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [47ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [86ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [87ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [90ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [90ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [180ms] refreshing cache from master, [181ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [184ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [186ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [187ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [188ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [662ms] refreshing cache from master, [662ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [666ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1480674920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23b17109 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642) -> (continuation of Deferred@1352224321 after retry RPC@361179164), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@581a19f6 -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642) -> (continuation of Deferred@1352224321 after retry RPC@361179164))) after lookup
08:07:28.620 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:28.622 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.622 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.622 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.775 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:28.776 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:28.776 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=825), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [3ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [27ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [29ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [29ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [47ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [86ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [87ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [90ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [90ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [180ms] refreshing cache from master, [181ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [184ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [186ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [187ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [188ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [662ms] refreshing cache from master, [662ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [666ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [668ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [671ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [671ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [820ms] refreshing cache from master, [821ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [824ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1480674920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23b17109 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642) -> (continuation of Deferred@1352224321 after retry RPC@361179164) -> (continuation of Deferred@1769392347 after retry RPC@1159584289), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@581a19f6 -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642) -> (continuation of Deferred@1352224321 after retry RPC@361179164) -> (continuation of Deferred@1769392347 after retry RPC@1159584289))) after lookup
08:07:28.777 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:28.778 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.779 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.779 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:28.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:28.900303 15577 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:29.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.021281 15593 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:29.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.021819 15593 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:29.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.024271 15593 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:29.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.028647 15401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:29.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.032729 15401 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:29.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.038117 15213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:29.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.039573 15398 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c; no voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0
08:07:29.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.040304 15593 raft_consensus.cc:2726] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
08:07:29.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.177884 15602 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:29.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.178426 15602 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:29.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.180105 15602 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.187312 15084 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:29.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.194325 15506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c" is_pre_election: true
08:07:29.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.195079 15506 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 4.
08:07:29.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.196498 15082 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.197232 15602 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 5 FOLLOWER]: Leader pre-election won for term 6
08:07:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.197582 15602 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:29.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.197902 15602 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 5 FOLLOWER]: Advancing to term 6
08:07:29.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.202281 15602 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:29.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.203923 15602 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 6 election: Requested vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:29.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.204806 15506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:29.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.205425 15506 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 4 FOLLOWER]: Advancing to term 6
08:07:29.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.207314 15084 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:29.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.212622 15506 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 6.
08:07:29.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.213644 15082 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:29.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.214259 15602 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Leader election won for term 6
08:07:29.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.215559 15602 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 LEADER]: Becoming Leader. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Running, Role: LEADER
08:07:29.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.216430 15602 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:29.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.225564 13651 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 reported cstate change: term changed from 5 to 6, leader changed from ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65) to ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67). New cstate: current_term: 6 leader_uuid: "ff4960fadb0549a59a1e29815dde2bb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: UNKNOWN } } }
08:07:29.316 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:29.317 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:29.317 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1366), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [3ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [27ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [29ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [29ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [47ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [86ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [87ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [90ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [90ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [180ms] refreshing cache from master, [181ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [184ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [186ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [187ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [188ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [662ms] refreshing cache from master, [662ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [666ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [668ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [671ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [671ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [820ms] refreshing cache from master, [821ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [824ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [826ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [828ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [828ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [1361ms] refreshing cache from master, [1361ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1365ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1480674920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@23b17109 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642) -> (continuation of Deferred@1352224321 after retry RPC@361179164) -> (continuation of Deferred@1769392347 after retry RPC@1159584289) -> (continuation of Deferred@802790157 after retry RPC@284155027), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@581a19f6 -> wakeup thread Time-limited test -> (continuation of Deferred@1054225280 after retry RPC@906713093) -> (continuation of Deferred@741328879 after retry RPC@881768315) -> (continuation of Deferred@931877751 after retry RPC@1988042642) -> (continuation of Deferred@1352224321 after retry RPC@361179164) -> (continuation of Deferred@1769392347 after retry RPC@1159584289) -> (continuation of Deferred@802790157 after retry RPC@284155027))) after lookup
08:07:29.319 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:29.321 [DEBUG - kudu-nio-0] (Connection.java:757) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] Successfully connected to peer
08:07:29.349 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x80f7cea3, L:/127.0.0.1:42778 - R:/127.12.255.67:43597] using TOKEN/null
08:07:29.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.361097 15506 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 6 FOLLOWER]: Refusing update from remote peer ff4960fadb0549a59a1e29815dde2bb0: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 18. (index mismatch)
08:07:29.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.361982 15084 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 -> Peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Couldn't send request to peer ec3940bcabac4a23852b5f27573cd9a1. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:29.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.362795 15610 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
08:07:29.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.390923 15630 mvcc.cc:205] Tried to move back new op lower bound from 7317594314163187712 to 7317594313598853120. Current Snapshot: MvccSnapshot[applied={T|T < 7317594314163187712}]
08:07:29.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.395183 15631 mvcc.cc:205] Tried to move back new op lower bound from 7317594314163187712 to 7317594313598853120. Current Snapshot: MvccSnapshot[applied={T|T < 7317594314163187712}]
08:07:29.444 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.65:38243
08:07:29.444 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.445753 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:29.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:29.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:29.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.787070 15639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:29.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.787710 15639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:29.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.788314 15639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:29.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.822813 15639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:29.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.823628 15639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.65
08:07:29.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.875329 15639 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:29.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:29.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:29.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.876787 15639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:29.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:29.878708 15639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:29.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.901079 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:30.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:30.378754 15084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111) [suppressed 4 similar messages]
08:07:31.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.901068 15654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:31.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:29.901320 15660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:31.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:31.133240 15659 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
08:07:31.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.133376 15639 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:07:31.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.137692 15639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:31.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.140496 15639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:31.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.141932 15639 hybrid_clock.cc:648] HybridClock initialized: now 1786522051141881 us; error 62 us; skew 500 ppm
08:07:31.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.142721 15639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:31.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.149446 15639 webserver.cc:460] Webserver started at http://127.12.255.65:46817/ using document root <none> and password file <none>
08:07:31.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.150650 15639 fs_manager.cc:359] Metadata directory not provided
08:07:31.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.150985 15639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:31.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.159621 15639 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
08:07:31.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.164664 15683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:31.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.166109 15639 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.166499 15639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:05 on dist-test-slave-ffrd"
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.168999 15639 fs_report.cc:383] FS layout report
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/data
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:31.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:31.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:31.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:31.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:31.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.220558 15639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:31.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.222281 15639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:31.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.222833 15639 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:31.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.225632 15639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:31.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.232147 15693 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:31.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.240913 15639 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:31.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.241245 15639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
08:07:31.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.241600 15639 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:31.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.247005 15639 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:31.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.247244 15639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.000s	sys 0.004s
08:07:31.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.247823 15693 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap starting.
08:07:31.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.316761 15693 log.cc:824] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Log is configured to *not* fsync() on all Append() calls
08:07:31.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.376451 15693 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:31.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.377516 15693 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap complete.
08:07:31.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.379343 15693 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent bootstrapping tablet: real 0.132s	user 0.113s	sys 0.013s
08:07:31.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.401399 15693 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:31.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.404364 15693 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Initialized, Role: FOLLOWER
08:07:31.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.405396 15693 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:31.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.409637 15693 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent starting tablet: real 0.030s	user 0.032s	sys 0.000s
08:07:31.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.443691 15639 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.65:38243
08:07:31.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.443974 15879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.65:38243 every 8 connection(s)
08:07:31.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.446326 15639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:31.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.449829 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 15639
08:07:31.450 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:31.527 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:31.530 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.67:43597
08:07:31.531 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:07:31.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.531668 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 15040
08:07:31.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.552204 15886 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:31.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.552783 15886 heartbeater.cc:464] Registering TS with master...
08:07:31.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.554386 15886 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:31.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.555109 15883 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:31.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.555497 15883 heartbeater.cc:464] Registering TS with master...
08:07:31.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.556335 15883 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:31.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.559938 13651 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:31.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.560035 13511 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:31.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.563710 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.65:34973
08:07:31.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.565915 15887 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:31.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.566255 15887 heartbeater.cc:464] Registering TS with master...
08:07:31.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.567338 15887 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:31.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:31.570724 13391 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:31.589 [DEBUG - kudu-nio-0] (Connection.java:239) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] handling channelInactive
08:07:31.590 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state READY due to: connection closed
08:07:31.590 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:31.591 [DEBUG - Time-limited test] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:31.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:31.591960 15901 negotiation.cc:337] Failed RPC negotiation. Trace:
08:07:31.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.463070 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.255.67:56625 (local address 127.12.255.65:38243)
08:07:31.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.483968 (+ 20898us) server_negotiation.cc:206] Beginning negotiation
08:07:31.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.484004 (+    36us) server_negotiation.cc:399] Waiting for connection header
08:07:31.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.497333 (+ 13329us) server_negotiation.cc:407] Connection header received
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.497816 (+   483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.497848 (+    32us) server_negotiation.cc:461] Received NEGOTIATE request from client
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.498269 (+   421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.498523 (+   254us) server_negotiation.cc:226] Negotiated authn=SASL
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.499821 (+  1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.502675 (+  2854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.507431 (+  4756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.523355 (+ 15924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.523697 (+   342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.527575 (+  3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.527614 (+    39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.527657 (+    43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.528882 (+  1225us) server_negotiation.cc:922] calling sasl_server_start()
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.546998 (+ 18116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.547080 (+    82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.547234 (+   154us) server_negotiation.cc:1035] Waiting for connection context
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:31.589198 (+ 41964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.255.67:56625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:07:31.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":1977,"server-negotiator.queue_time_us":17076,"thread_start_us":17698,"threads_started":1}
08:07:31.593 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.593 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.594 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.616 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:31.616 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:31.617 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1691016772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2fca7960 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10482bb1 -> wakeup thread Time-limited test)) after lookup
08:07:31.617 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:31.619 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.619 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.620 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.636 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:31.637 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:31.637 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [28ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1691016772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2fca7960 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10482bb1 -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409))) after lookup
08:07:31.638 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:31.639 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.639 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.639 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.676 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:31.676 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:31.677 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [28ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1691016772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2fca7960 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10482bb1 -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061))) after lookup
08:07:31.677 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:31.679 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.679 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.679 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.696 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:31.696 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:31.696 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=105), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [28ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [86ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [88ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [88ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [104ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1691016772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2fca7960 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061) -> (continuation of Deferred@545583085 after retry RPC@1501326900), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10482bb1 -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061) -> (continuation of Deferred@545583085 after retry RPC@1501326900))) after lookup
08:07:31.698 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:31.698 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.699 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:31.699 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:32.016 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:32.016 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:32.016 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=425), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [28ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [86ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [88ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [88ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [104ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [106ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [108ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [108ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [421ms] refreshing cache from master, [421ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [424ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1691016772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2fca7960 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061) -> (continuation of Deferred@545583085 after retry RPC@1501326900) -> (continuation of Deferred@1721433584 after retry RPC@831971227), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10482bb1 -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061) -> (continuation of Deferred@545583085 after retry RPC@1501326900) -> (continuation of Deferred@1721433584 after retry RPC@831971227))) after lookup
08:07:32.017 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:32.019 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:32.019 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:32.020 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:32.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.568373 15883 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:32.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.653422 15912 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff4960fadb0549a59a1e29815dde2bb0)
08:07:32.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.653965 15912 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:32.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.655620 15912 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:32.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:32.661490 15398 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111)
08:07:32.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.669387 15790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1" is_pre_election: true
08:07:32.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.670104 15790 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e631edcb842e4e809e7bc9f4a63a388c in term 5.
08:07:32.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.671409 15401 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: ff4960fadb0549a59a1e29815dde2bb0
08:07:32.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.672093 15912 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 6 FOLLOWER]: Leader pre-election won for term 7
08:07:32.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.672448 15912 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff4960fadb0549a59a1e29815dde2bb0)
08:07:32.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.672797 15912 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 6 FOLLOWER]: Advancing to term 7
08:07:32.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.677488 15912 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:32.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.679076 15912 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 7 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:32.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.680058 15790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:32.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.680635 15790 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 5 FOLLOWER]: Advancing to term 7
08:07:32.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:32.682003 15398 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111)
08:07:32.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.688683 15790 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e631edcb842e4e809e7bc9f4a63a388c in term 7.
08:07:32.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.689692 15401 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: ff4960fadb0549a59a1e29815dde2bb0
08:07:32.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.690308 15912 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Leader election won for term 7
08:07:32.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.691412 15912 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 LEADER]: Becoming Leader. State: Replica: e631edcb842e4e809e7bc9f4a63a388c, State: Running, Role: LEADER
08:07:32.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.692109 15912 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:32.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:32.700410 13651 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c reported cstate change: term changed from 6 to 7, leader changed from ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67) to e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66). New cstate: current_term: 7 leader_uuid: "e631edcb842e4e809e7bc9f4a63a388c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: UNKNOWN } } }
08:07:33.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:33.220867 15398 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c -> Peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Couldn't send request to peer ff4960fadb0549a59a1e29815dde2bb0. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:33.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:33.261529 15790 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 7 FOLLOWER]: Refusing update from remote peer e631edcb842e4e809e7bc9f4a63a388c: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
08:07:33.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:33.263237 15940 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
08:07:33.836 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:33.836 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:33.836 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2245), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [25ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [26ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [28ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [29ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [45ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [46ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [48ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [48ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [86ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [88ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [88ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [104ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [106ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [108ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [108ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [421ms] refreshing cache from master, [421ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [424ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [426ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [428ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [429ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [2241ms] refreshing cache from master, [2241ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [2244ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1691016772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@2fca7960 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061) -> (continuation of Deferred@545583085 after retry RPC@1501326900) -> (continuation of Deferred@1721433584 after retry RPC@831971227) -> (continuation of Deferred@78047228 after retry RPC@55286516), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@10482bb1 -> wakeup thread Time-limited test -> (continuation of Deferred@629321275 after retry RPC@1959136409) -> (continuation of Deferred@1735312365 after retry RPC@1181220061) -> (continuation of Deferred@545583085 after retry RPC@1501326900) -> (continuation of Deferred@1721433584 after retry RPC@831971227) -> (continuation of Deferred@78047228 after retry RPC@55286516))) after lookup
08:07:33.837 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:33.838 [DEBUG - kudu-nio-1] (Connection.java:757) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] Successfully connected to peer
08:07:33.883 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x129b5a56, L:/127.0.0.1:41696 - R:/127.12.255.66:39593] using TOKEN/null
08:07:33.929 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.67:43597
08:07:33.931 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:33.932633 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:43597
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37217
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:33.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:33.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:33.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:33.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:34.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.188318 15398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111) [suppressed 4 similar messages]
08:07:34.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.259021 15981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:34.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.259485 15981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:34.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.259924 15981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:34.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.293744 15981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:34.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.294701 15981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.67
08:07:34.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:34.336886 15981 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:34.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:34.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:43597
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37217
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:34.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:34.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:34.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:34.338395 15981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:34.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:34.340371 15981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:34.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.360529 15994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:35.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:34.363019 15981 server_base.cc:1034] running on GCE node
08:07:35.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.360821 15991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:35.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:34.360852 15992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:35.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.637555 15981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:35.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.639943 15981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:35.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.641350 15981 hybrid_clock.cc:648] HybridClock initialized: now 1786522055641303 us; error 46 us; skew 500 ppm
08:07:35.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.642177 15981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:35.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.653203 15981 webserver.cc:460] Webserver started at http://127.12.255.67:37217/ using document root <none> and password file <none>
08:07:35.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.654304 15981 fs_manager.cc:359] Metadata directory not provided
08:07:35.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.654548 15981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:35.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.662926 15981 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.004s
08:07:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.668099 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.669365 15981 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:07:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.669761 15981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:09 on dist-test-slave-ffrd"
08:07:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.672403 15981 fs_report.cc:383] FS layout report
08:07:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/data
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:35.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:35.684002 15398 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c -> Peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Couldn't send request to peer ff4960fadb0549a59a1e29815dde2bb0. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:07:35.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.718878 15981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:35.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.720531 15981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:35.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.721104 15981 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.723861 15981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:35.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.729974 16023 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:35.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.738929 15981 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:35.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.739207 15981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
08:07:35.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.739581 15981 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:35.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.745319 15981 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:35.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.745587 15981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.005s	sys 0.001s
08:07:35.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.746031 16023 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap starting.
08:07:35.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.827595 16023 log.cc:824] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Log is configured to *not* fsync() on all Append() calls
08:07:35.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.927917 16023 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:35.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.929172 16023 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap complete.
08:07:35.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.930673 16023 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent bootstrapping tablet: real 0.185s	user 0.126s	sys 0.050s
08:07:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.953794 16023 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:35.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.957005 16023 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Initialized, Role: FOLLOWER
08:07:35.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.958984 16023 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:35.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.962888 16023 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent starting tablet: real 0.032s	user 0.026s	sys 0.005s
08:07:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.970882 15981 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.67:43597
08:07:35.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.971084 16204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.67:43597 every 8 connection(s)
08:07:35.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.973582 15981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:35.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:35.978513 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 15981
08:07:35.979 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:35.986 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
08:07:35.988 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
08:07:36.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.023768 16205 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:36.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.024235 16205 heartbeater.cc:464] Registering TS with master...
08:07:36.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.025879 16205 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:36.027 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:36.028 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.66:39593
08:07:36.028 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:07:36.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.029035 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 15360
08:07:36.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.040510 13651 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:36.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.044037 16207 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:36.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.044437 16209 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:36.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.044451 16207 heartbeater.cc:464] Registering TS with master...
08:07:36.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.044795 16209 heartbeater.cc:464] Registering TS with master...
08:07:36.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.045780 16207 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:36.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.045708 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.67:48243
08:07:36.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.046974 16209 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:36.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.049631 13511 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:36.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:36.051748 13391 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:36.082 [DEBUG - kudu-nio-1] (Connection.java:239) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] handling channelInactive
08:07:36.082 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state READY due to: connection closed
08:07:36.084 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:36.085 [DEBUG - Time-limited test] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:36.087 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.087 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.088 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.097 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:36.097 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:36.097 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test)) after lookup
08:07:36.098 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:36.099 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.099 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.099 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.117 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:36.117 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:36.118 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [14ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [15ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [15ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [32ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733))) after lookup
08:07:36.118 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:36.120 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.120 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.121 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.157 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:36.157 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:36.157 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [14ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [15ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [15ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [32ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [34ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [36ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [37ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [72ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554))) after lookup
08:07:36.158 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:36.159 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.159 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.159 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.176 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:36.176 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:36.177 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=92), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [14ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [15ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [15ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [32ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [34ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [36ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [37ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [72ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [74ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [75ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [75ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [92ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459))) after lookup
08:07:36.177 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:36.179 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.180 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.180 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.236 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:36.237 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:36.238 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=153), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [14ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [15ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [15ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [32ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [34ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [36ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [37ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [72ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [74ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [75ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [75ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [92ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [93ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [96ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [96ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [152ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821))) after lookup
08:07:36.238 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:36.239 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.240 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.240 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.976 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:36.977 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:36.977 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=892), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [14ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [15ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [15ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [32ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [34ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [36ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [37ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [72ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [74ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [75ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [75ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [92ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [93ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [96ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [96ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [152ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [154ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [156ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [156ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [888ms] refreshing cache from master, [888ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [892ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821) -> (continuation of Deferred@624013284 after retry RPC@899167042), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821) -> (continuation of Deferred@624013284 after retry RPC@899167042))) after lookup
08:07:36.978 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:36.980 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.980 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:36.981 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:37.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.052613 16205 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:37.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.121102 16225 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:37.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.121551 16225 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:37.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.123773 16225 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:37.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:37.127799 16010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:37.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:37.131760 16010 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:37.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.136493 15790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1" is_pre_election: true
08:07:37.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.137991 16015 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1
08:07:37.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.138641 16225 raft_consensus.cc:2726] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
08:07:37.416 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:37.417 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:37.417 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1332), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [14ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [15ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [15ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [32ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [34ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [36ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [37ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [72ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [74ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [75ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [75ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [92ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [93ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [96ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [96ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [152ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [154ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [156ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [156ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [888ms] refreshing cache from master, [888ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [892ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [894ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [897ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [897ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [1328ms] refreshing cache from master, [1328ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1332ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821) -> (continuation of Deferred@624013284 after retry RPC@899167042) -> (continuation of Deferred@180744267 after retry RPC@2125713522), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821) -> (continuation of Deferred@624013284 after retry RPC@899167042) -> (continuation of Deferred@180744267 after retry RPC@2125713522))) after lookup
08:07:37.418 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:37.419 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:37.419 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:37.419 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:37.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.424772 16233 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e631edcb842e4e809e7bc9f4a63a388c)
08:07:37.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.425199 16233 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:37.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.427330 16233 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:37.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:37.431639 15686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:37.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:37.435467 15686 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:37.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.441428 16133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:37.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.442222 16133 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 6.
08:07:37.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.443565 15687 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c
08:07:37.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.444378 16233 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 7 FOLLOWER]: Leader pre-election won for term 8
08:07:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.444739 16233 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e631edcb842e4e809e7bc9f4a63a388c)
08:07:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.445142 16233 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 7 FOLLOWER]: Advancing to term 8
08:07:37.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.449615 16233 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:37.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.451274 16233 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 8 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:37.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.452047 16133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:37.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.452647 16133 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 6 FOLLOWER]: Advancing to term 8
08:07:37.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:37.459818 15686 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.462568 16133 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 8.
08:07:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.463559 15687 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c
08:07:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.464126 16233 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 8 FOLLOWER]: Leader election won for term 8
08:07:37.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.465696 16233 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 8 LEADER]: Becoming Leader. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Running, Role: LEADER
08:07:37.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.466598 16233 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:37.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.475411 13651 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 reported cstate change: term changed from 7 to 8, leader changed from e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66) to ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65). New cstate: current_term: 8 leader_uuid: "ec3940bcabac4a23852b5f27573cd9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: HEALTHY } } }
08:07:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.930227 16133 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 8 FOLLOWER]: Refusing update from remote peer ec3940bcabac4a23852b5f27573cd9a1: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
08:07:37.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:37.931814 16233 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
08:07:37.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:37.980818 15686 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 -> Peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Couldn't send request to peer e631edcb842e4e809e7bc9f4a63a388c. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:40.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:40.598773 15686 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 -> Peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Couldn't send request to peer e631edcb842e4e809e7bc9f4a63a388c. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:07:41.157 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:41.157 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:41.157 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=16, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=5073), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [3ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [4ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [12ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [14ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [15ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [15ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [32ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [34ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [36ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [37ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [72ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [74ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [75ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [75ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [92ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [93ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [96ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [96ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [152ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [154ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [156ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [156ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [888ms] refreshing cache from master, [888ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [892ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [894ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [897ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [897ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [1328ms] refreshing cache from master, [1328ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1332ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [1333ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [1335ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [1335ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [5068ms] refreshing cache from master, [5068ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [5072ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@255665317(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@e1f3f21 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821) -> (continuation of Deferred@624013284 after retry RPC@899167042) -> (continuation of Deferred@180744267 after retry RPC@2125713522) -> (continuation of Deferred@175921686 after retry RPC@1165513798), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7b378b63 -> wakeup thread Time-limited test -> (continuation of Deferred@1671804538 after retry RPC@1786082733) -> (continuation of Deferred@1012578857 after retry RPC@2101569554) -> (continuation of Deferred@564583092 after retry RPC@252088459) -> (continuation of Deferred@1362497936 after retry RPC@1779384821) -> (continuation of Deferred@624013284 after retry RPC@899167042) -> (continuation of Deferred@180744267 after retry RPC@2125713522) -> (continuation of Deferred@175921686 after retry RPC@1165513798))) after lookup
08:07:41.158 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:41.160 [DEBUG - kudu-nio-0] (Connection.java:757) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] Successfully connected to peer
08:07:41.179 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x3d56ddf5, L:/127.0.0.1:41244 - R:/127.12.255.65:38243] using TOKEN/null
08:07:41.231 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.66:39593
08:07:41.231 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:41.234174 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:41.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:39593
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42867
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:41.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:41.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.566409 16319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:41.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.566895 16319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:41.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.567338 16319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:41.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.599570 16319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:41.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.600395 16319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.66
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:41.635821 16319 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:39593
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42867
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:41.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:41.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:41.637179 16319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:41.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:41.638754 16319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:41.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.657825 16330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:42.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:42.650723 15686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111) [suppressed 10 similar messages]
08:07:42.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.657945 16334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:42.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:41.657905 16329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:42.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:42.993324 16333 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1334 milliseconds
08:07:42.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:42.993448 16319 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:07:43.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.000550 16319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:43.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.009974 16319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.011529 16319 hybrid_clock.cc:648] HybridClock initialized: now 1786522063011473 us; error 46 us; skew 500 ppm
08:07:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.012370 16319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:43.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.019197 16319 webserver.cc:460] Webserver started at http://127.12.255.66:42867/ using document root <none> and password file <none>
08:07:43.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.020318 16319 fs_manager.cc:359] Metadata directory not provided
08:07:43.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.020529 16319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:43.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.028697 16319 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.004s
08:07:43.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.033396 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:43.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.034382 16319 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.034772 16319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:08 on dist-test-slave-ffrd"
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.037221 16319 fs_report.cc:383] FS layout report
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:43.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/data
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:43.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:43.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.087188 16319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:43.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.089282 16319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:43.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.089825 16319 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:43.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.093170 16319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:43.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.101466 16360 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:43.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.110309 16319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:43.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.110608 16319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.003s	sys 0.000s
08:07:43.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.110934 16319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:43.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.116642 16319 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:43.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.116822 16319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.003s	sys 0.000s
08:07:43.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.117308 16360 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap starting.
08:07:43.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.179556 16360 log.cc:824] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Log is configured to *not* fsync() on all Append() calls
08:07:43.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.264191 16360 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:43.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.265257 16360 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap complete.
08:07:43.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.267164 16360 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent bootstrapping tablet: real 0.150s	user 0.103s	sys 0.043s
08:07:43.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:43.269230 15686 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 -> Peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Couldn't send request to peer e631edcb842e4e809e7bc9f4a63a388c. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
08:07:43.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.289336 16360 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:43.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.292681 16360 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: e631edcb842e4e809e7bc9f4a63a388c, State: Initialized, Role: FOLLOWER
08:07:43.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.293919 16360 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:43.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.300309 16360 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent starting tablet: real 0.033s	user 0.033s	sys 0.000s
08:07:43.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.312160 16319 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.66:39593
08:07:43.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.312326 16531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.66:39593 every 8 connection(s)
08:07:43.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.314903 16319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:43.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.315763 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 16319
08:07:43.317 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:43.348 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:43.349 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.65:38243
08:07:43.349 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:43.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.350890 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 15639
08:07:43.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.377959 16533 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:43.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.378644 16533 heartbeater.cc:464] Registering TS with master...
08:07:43.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.380242 16533 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:43.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.385021 13511 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:43.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.389928 16534 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:43.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.390308 16534 heartbeater.cc:464] Registering TS with master...
08:07:43.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.391237 16534 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.392927 16532 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:43.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.393471 16532 heartbeater.cc:464] Registering TS with master...
08:07:43.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.394411 16532 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:43.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.397571 13391 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:43.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.397607 13651 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:43.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:43.400374 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.66:53213
08:07:43.412 [DEBUG - kudu-nio-0] (Connection.java:239) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] handling channelInactive
08:07:43.412 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state READY due to: connection closed
08:07:43.413 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:43.414 [DEBUG - Time-limited test] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:43.415 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.415 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.415 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.436 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:43.436 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:43.436 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [22ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1353532185(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@378310b8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ab2f0a3 -> wakeup thread Time-limited test)) after lookup
08:07:43.437 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:43.439 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.439 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.439 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.456 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:43.457 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:43.457 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [22ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [24ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [26ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [26ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1353532185(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@378310b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ab2f0a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823))) after lookup
08:07:43.458 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:43.459 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.459 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.460 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.498 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:43.499 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:43.499 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [22ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [24ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [26ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [26ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [46ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1353532185(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@378310b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ab2f0a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557))) after lookup
08:07:43.500 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:43.502 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.502 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.502 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.596 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:43.596 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:43.597 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=183), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [22ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [24ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [26ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [26ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [46ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [86ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [89ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [89ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [182ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1353532185(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@378310b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557) -> (continuation of Deferred@1650455933 after retry RPC@109811685), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ab2f0a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557) -> (continuation of Deferred@1650455933 after retry RPC@109811685))) after lookup
08:07:43.597 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:43.598 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.599 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:43.599 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:44.076 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:44.076 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:44.077 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=663), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [22ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [24ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [26ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [26ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [46ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [86ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [89ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [89ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [182ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [184ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [186ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [186ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [658ms] refreshing cache from master, [659ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [662ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1353532185(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@378310b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557) -> (continuation of Deferred@1650455933 after retry RPC@109811685) -> (continuation of Deferred@1371451652 after retry RPC@2110008334), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ab2f0a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557) -> (continuation of Deferred@1650455933 after retry RPC@109811685) -> (continuation of Deferred@1371451652 after retry RPC@2110008334))) after lookup
08:07:44.078 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:44.080 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:44.080 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:44.080 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:44.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.404148 16532 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:44.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.443660 16555 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:44.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.444243 16555 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:44.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.446696 16555 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:44.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:44.451438 16355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:44.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:44.456038 16355 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:44.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.461468 16133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:44.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.463269 16354 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c; no voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0
08:07:44.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.464002 16555 raft_consensus.cc:2726] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
08:07:44.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.606391 16566 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:44.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.606966 16566 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:44.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.609071 16566 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:44.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:44.616281 16015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:44.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:44.620086 16015 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:44.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.625084 16461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c" is_pre_election: true
08:07:44.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.625767 16461 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 7.
08:07:44.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.627017 16010 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:44.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.627607 16566 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 8 FOLLOWER]: Leader pre-election won for term 9
08:07:44.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.627991 16566 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:44.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.628334 16566 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 8 FOLLOWER]: Advancing to term 9
08:07:44.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.633507 16566 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:44.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.634887 16566 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 9 election: Requested vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:44.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.635877 16461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:44.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.636425 16461 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 7 FOLLOWER]: Advancing to term 9
08:07:44.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:44.638057 16015 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:44.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.643285 16461 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 9.
08:07:44.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.644243 16010 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:44.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.644902 16566 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 9 FOLLOWER]: Leader election won for term 9
08:07:44.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.646100 16566 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 9 LEADER]: Becoming Leader. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Running, Role: LEADER
08:07:44.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.647558 16566 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:44.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:44.657374 13651 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 reported cstate change: term changed from 8 to 9, leader changed from ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65) to ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67). New cstate: current_term: 9 leader_uuid: "ff4960fadb0549a59a1e29815dde2bb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: UNKNOWN } } }
08:07:45.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:45.222856 16015 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 -> Peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Couldn't send request to peer ec3940bcabac4a23852b5f27573cd9a1. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:45.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:45.246779 16461 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 9 FOLLOWER]: Refusing update from remote peer ff4960fadb0549a59a1e29815dde2bb0: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
08:07:45.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:45.248224 16598 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
08:07:45.736 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:45.736 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:45.737 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2323), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [22ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [24ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [26ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [26ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [43ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [46ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [47ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [85ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [86ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [89ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [89ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [182ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [184ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [186ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [186ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [658ms] refreshing cache from master, [659ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [662ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [664ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [667ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [667ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2319ms] refreshing cache from master, [2319ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [2322ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1353532185(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@378310b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557) -> (continuation of Deferred@1650455933 after retry RPC@109811685) -> (continuation of Deferred@1371451652 after retry RPC@2110008334) -> (continuation of Deferred@2008031754 after retry RPC@734577838), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ab2f0a3 -> wakeup thread Time-limited test -> (continuation of Deferred@1210951545 after retry RPC@1167552823) -> (continuation of Deferred@1882288673 after retry RPC@1627436557) -> (continuation of Deferred@1650455933 after retry RPC@109811685) -> (continuation of Deferred@1371451652 after retry RPC@2110008334) -> (continuation of Deferred@2008031754 after retry RPC@734577838))) after lookup
08:07:45.738 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:45.739 [DEBUG - kudu-nio-1] (Connection.java:757) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] Successfully connected to peer
08:07:45.763 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x41493919, L:/127.0.0.1:36110 - R:/127.12.255.67:43597] using TOKEN/null
08:07:45.820 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.65:38243
08:07:45.821 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:45.822916 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:45.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:46.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.148581 16630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:46.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.149099 16630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:46.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.149536 16630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:46.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.183985 16630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:46.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.184828 16630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.65
08:07:46.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:46.223428 16630 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:46.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:46.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:46.224749 16630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:46.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:46.226356 16630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:46.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.245348 16645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:47.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:47.444396 16015 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 -> Peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Couldn't send request to peer ec3940bcabac4a23852b5f27573cd9a1. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:07:47.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.249028 16647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:47.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:46.245496 16652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:47.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:47.497190 16649 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
08:07:47.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.497334 16630 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:07:47.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.498692 16630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:47.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.501296 16630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:47.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.502755 16630 hybrid_clock.cc:648] HybridClock initialized: now 1786522067502729 us; error 43 us; skew 500 ppm
08:07:47.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.503723 16630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:47.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.515367 16630 webserver.cc:460] Webserver started at http://127.12.255.65:46817/ using document root <none> and password file <none>
08:07:47.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.516451 16630 fs_manager.cc:359] Metadata directory not provided
08:07:47.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.516706 16630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:47.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.525067 16630 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.004s
08:07:47.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.529655 16662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:47.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.530990 16630 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.004s
08:07:47.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.531388 16630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:05 on dist-test-slave-ffrd"
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.533968 16630 fs_report.cc:383] FS layout report
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/data
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:47.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:47.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:47.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:47.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.578969 16630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:47.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.580552 16630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:47.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.581126 16630 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:47.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.583773 16630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:47.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.590967 16679 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:47.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.599387 16630 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:47.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.599622 16630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.003s	sys 0.000s
08:07:47.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.599825 16630 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:47.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.604619 16630 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:47.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.604878 16630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.000s	sys 0.002s
08:07:47.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.605286 16679 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap starting.
08:07:47.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.689127 16679 log.cc:824] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Log is configured to *not* fsync() on all Append() calls
08:07:47.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.774170 16679 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:47.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.775374 16679 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap complete.
08:07:47.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.777741 16679 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent bootstrapping tablet: real 0.173s	user 0.109s	sys 0.059s
08:07:47.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.810294 16630 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.65:38243
08:07:47.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.810480 16867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.65:38243 every 8 connection(s)
08:07:47.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.809044 16679 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:47.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.812708 16679 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Initialized, Role: FOLLOWER
08:07:47.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.814131 16630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:47.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.813952 16679 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.818935 16679 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent starting tablet: real 0.041s	user 0.039s	sys 0.001s
08:07:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.820139 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 16630
08:07:47.821 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:47.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.853210 16871 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:47.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.853719 16871 heartbeater.cc:464] Registering TS with master...
08:07:47.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.854939 16871 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:47.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.860909 13651 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:47.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.864049 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.65:52229
08:07:47.872 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:47.873 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.67:43597
08:07:47.873 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:07:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.873824 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 15981
08:07:47.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.896919 16872 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:47.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.897377 16872 heartbeater.cc:464] Registering TS with master...
08:07:47.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.898239 16872 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:47.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.902799 13512 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:47.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.916633 16883 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:47.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.917004 16883 heartbeater.cc:464] Registering TS with master...
08:07:47.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.917716 16883 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:47.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:47.921308 13390 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:47.931 [DEBUG - kudu-nio-1] (Connection.java:239) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] handling channelInactive
08:07:47.931 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state READY due to: connection closed
08:07:47.932 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:47.933 [DEBUG - Time-limited test] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:47.935 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.936 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.936 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.956 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:47.957 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:47.957 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [23ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1419432052(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@72cb9b04 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3409e0fa -> wakeup thread Time-limited test)) after lookup
08:07:47.958 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:47.959 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.959 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.959 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:47.975 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:47.976 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [23ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [26ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [26ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [42ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1419432052(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@72cb9b04 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3409e0fa -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770))) after lookup
08:07:47.976 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:47.978 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.978 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:47.978 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.015 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:48.016 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:48.016 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [23ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [26ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [26ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [42ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [43ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [45ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [45ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [82ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1419432052(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@72cb9b04 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3409e0fa -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237))) after lookup
08:07:48.017 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:48.018 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.018 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.018 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.156 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:48.156 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:48.157 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=223), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [23ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [26ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [26ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [42ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [43ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [45ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [45ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [82ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [83ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [85ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [85ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [219ms] refreshing cache from master, [219ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [222ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1419432052(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@72cb9b04 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237) -> (continuation of Deferred@1317321746 after retry RPC@2118777720), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3409e0fa -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237) -> (continuation of Deferred@1317321746 after retry RPC@2118777720))) after lookup
08:07:48.157 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:48.159 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.159 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.159 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.175 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:48.176 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:48.176 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=243), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [23ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [26ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [26ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [42ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [43ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [45ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [45ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [82ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [83ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [85ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [85ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [219ms] refreshing cache from master, [219ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [222ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [224ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [226ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [226ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [239ms] refreshing cache from master, [239ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [242ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1419432052(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@72cb9b04 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237) -> (continuation of Deferred@1317321746 after retry RPC@2118777720) -> (continuation of Deferred@1555874678 after retry RPC@1830773154), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3409e0fa -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237) -> (continuation of Deferred@1317321746 after retry RPC@2118777720) -> (continuation of Deferred@1555874678 after retry RPC@1830773154))) after lookup
08:07:48.177 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:48.178 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.178 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state CONNECTING due to: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.178 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597
08:07:48.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:48.870092 16871 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:49.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.360563 16889 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff4960fadb0549a59a1e29815dde2bb0)
08:07:49.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.361085 16889 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:49.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.362671 16889 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:49.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:49.369906 16354 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111)
08:07:49.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.377039 16785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1" is_pre_election: true
08:07:49.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.377748 16785 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e631edcb842e4e809e7bc9f4a63a388c in term 8.
08:07:49.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.378899 16355 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: ff4960fadb0549a59a1e29815dde2bb0
08:07:49.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.379487 16889 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 9 FOLLOWER]: Leader pre-election won for term 10
08:07:49.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.379724 16889 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff4960fadb0549a59a1e29815dde2bb0)
08:07:49.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.379976 16889 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 9 FOLLOWER]: Advancing to term 10
08:07:49.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.384251 16889 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:49.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.385820 16889 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 10 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:49.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.386900 16785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:07:49.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.387475 16785 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 8 FOLLOWER]: Advancing to term 10
08:07:49.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:49.389043 16354 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111)
08:07:49.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.394630 16785 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e631edcb842e4e809e7bc9f4a63a388c in term 10.
08:07:49.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.395744 16355 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ec3940bcabac4a23852b5f27573cd9a1; no voters: ff4960fadb0549a59a1e29815dde2bb0
08:07:49.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.396394 16889 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Leader election won for term 10
08:07:49.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.397461 16889 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 LEADER]: Becoming Leader. State: Replica: e631edcb842e4e809e7bc9f4a63a388c, State: Running, Role: LEADER
08:07:49.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.398164 16889 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:49.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.406813 13651 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c reported cstate change: term changed from 9 to 10, leader changed from ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67) to e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66). New cstate: current_term: 10 leader_uuid: "e631edcb842e4e809e7bc9f4a63a388c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: UNKNOWN } } }
08:07:49.636 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:49.636 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:49.637 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1703), Traces: [0ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [3ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [3ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [23ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [26ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [26ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [42ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [43ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [45ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [45ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [82ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [83ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [85ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [85ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [219ms] refreshing cache from master, [219ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [222ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [224ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [226ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [226ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [239ms] refreshing cache from master, [239ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [242ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [244ms] sending RPC to server ff4960fadb0549a59a1e29815dde2bb0, [245ms] received response from server ff4960fadb0549a59a1e29815dde2bb0: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [245ms] delaying RPC due to: Network error: Failed to connect to peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597): Connection refused: /127.12.255.67:43597, [1699ms] refreshing cache from master, [1699ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1702ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@1419432052(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@72cb9b04 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237) -> (continuation of Deferred@1317321746 after retry RPC@2118777720) -> (continuation of Deferred@1555874678 after retry RPC@1830773154) -> (continuation of Deferred@1932955917 after retry RPC@1595264568), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3409e0fa -> wakeup thread Time-limited test -> (continuation of Deferred@1417292291 after retry RPC@1577698770) -> (continuation of Deferred@151776091 after retry RPC@431786237) -> (continuation of Deferred@1317321746 after retry RPC@2118777720) -> (continuation of Deferred@1555874678 after retry RPC@1830773154) -> (continuation of Deferred@1932955917 after retry RPC@1595264568))) after lookup
08:07:49.637 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:49.639 [DEBUG - kudu-nio-0] (Connection.java:757) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] Successfully connected to peer
08:07:49.660 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x412a8cc5, L:/127.0.0.1:38686 - R:/127.12.255.66:39593] using TOKEN/null
08:07:49.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:49.669612 16354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111) [suppressed 2 similar messages]
08:07:49.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.669806 16785 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 10 FOLLOWER]: Refusing update from remote peer e631edcb842e4e809e7bc9f4a63a388c: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 34. (index mismatch)
08:07:49.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.671489 16900 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.001s
08:07:49.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:49.672173 16354 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c -> Peer ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597): Couldn't send request to peer ff4960fadb0549a59a1e29815dde2bb0. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.67:43597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.697919 16924 mvcc.cc:205] Tried to move back new op lower bound from 7317594397352689664 to 7317594396263256064. Current Snapshot: MvccSnapshot[applied={T|T < 7317594397352689664}]
08:07:49.734 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.67:43597
08:07:49.734 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:49.736038 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:49.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:43597
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37217
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:49.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:50.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.085908 16935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:50.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.086508 16935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:50.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.087119 16935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:50.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.122490 16935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:50.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.123486 16935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.67
08:07:50.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:50.162415 16935 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.67:43597
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.67
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37217
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.67
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/logs
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:50.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:50.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:50.164060 16935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:50.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:50.165764 16935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:50.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.185262 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.185290 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:50.185670 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:51.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:50.187749 16935 server_base.cc:1034] running on GCE node
08:07:51.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.407436 16935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:51.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.410600 16935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:51.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.412082 16935 hybrid_clock.cc:648] HybridClock initialized: now 1786522071412053 us; error 53 us; skew 500 ppm
08:07:51.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.412919 16935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:51.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.420271 16935 webserver.cc:460] Webserver started at http://127.12.255.67:37217/ using document root <none> and password file <none>
08:07:51.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.421339 16935 fs_manager.cc:359] Metadata directory not provided
08:07:51.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.421561 16935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:51.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.430136 16935 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.001s
08:07:51.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.436251 16965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:51.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.437525 16935 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.005s
08:07:51.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.437821 16935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:51.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:51.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:09 on dist-test-slave-ffrd"
08:07:51.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.440382 16935 fs_report.cc:383] FS layout report
08:07:51.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:51.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:51.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/wal
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/data
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:51.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:51.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.485476 16935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:51.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.487697 16935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:51.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.488237 16935 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:51.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.491468 16935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:51.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.497740 16976 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:51.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.506022 16935 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:51.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.506297 16935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
08:07:51.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.506546 16935 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:51.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.511844 16935 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:51.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.512024 16935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
08:07:51.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.512481 16976 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap starting.
08:07:51.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.594458 16976 log.cc:824] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Log is configured to *not* fsync() on all Append() calls
08:07:51.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.698145 16935 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.67:43597
08:07:51.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.698324 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.67:43597 every 8 connection(s)
08:07:51.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.701975 16935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-2/data/info.pb
08:07:51.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.712389 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 16935
08:07:51.713 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:51.772 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:51.773 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.66:39593
08:07:51.773 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:07:51.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.773934 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 16319
08:07:51.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.811697 16976 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:51.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.813364 16976 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Bootstrap complete.
08:07:51.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.816443 16976 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent bootstrapping tablet: real 0.304s	user 0.159s	sys 0.065s
08:07:51.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.825156 17181 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:51.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.825552 17180 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:51.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.825665 17181 heartbeater.cc:464] Registering TS with master...
08:07:51.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.825838 17180 heartbeater.cc:464] Registering TS with master...
08:07:51.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.826951 17180 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.829273 17181 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:51.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.832758 13651 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:51.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.833290 13512 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:51.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.836328 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.67:59975
08:07:51.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.844904 16976 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:51.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.849406 16976 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Initialized, Role: FOLLOWER
08:07:51.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.850795 16976 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:51.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.878988 17180 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:51.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.879694 16976 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0: Time spent starting tablet: real 0.063s	user 0.041s	sys 0.020s
08:07:51.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.880822 17185 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:51.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.881254 17185 heartbeater.cc:464] Registering TS with master...
08:07:51.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.882031 17185 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:51.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:51.887090 13390 ts_manager.cc:194] Re-registered known tserver with Master: ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597)
08:07:51.909 [DEBUG - kudu-nio-0] (Connection.java:239) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] handling channelInactive
08:07:51.909 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state READY due to: connection closed
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:51.910967 17176 negotiation.cc:337] Failed RPC negotiation. Trace:
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.701763 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.255.66:60347 (local address 127.12.255.67:43597)
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.705472 (+  3709us) server_negotiation.cc:206] Beginning negotiation
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.705517 (+    45us) server_negotiation.cc:399] Waiting for connection header
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.705591 (+    74us) server_negotiation.cc:407] Connection header received
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.716406 (+ 10815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.716467 (+    61us) server_negotiation.cc:461] Received NEGOTIATE request from client
08:07:51.911 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.717134 (+   667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.722160 (+  5026us) server_negotiation.cc:226] Negotiated authn=SASL
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.728637 (+  6477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.759085 (+ 30448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.762574 (+  3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.764156 (+  1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.771587 (+  7431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:07:51.909278 (+137691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.255.66:60347: BlockingRecv error: recv got EOF from 127.12.255.66:60347 (error 108)
08:07:51.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2901,"thread_start_us":2422,"threads_started":1}
08:07:51.912 [DEBUG - Time-limited test] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:51.914 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.914 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.914 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.936 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:51.936 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:51.937 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@259218191(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@6f703b29 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25271bf -> wakeup thread Time-limited test)) after lookup
08:07:51.937 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:51.939 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.939 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.939 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.956 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:51.957 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:51.957 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [27ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [27ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [44ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@259218191(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@6f703b29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25271bf -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365))) after lookup
08:07:51.957 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:51.958 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.958 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.959 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:51.997 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:51.997 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:51.998 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=86), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [27ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [27ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [44ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [47ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [47ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [84ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@259218191(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@6f703b29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25271bf -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611))) after lookup
08:07:51.999 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:52.001 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.001 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.001 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.136 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:52.137 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:52.137 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=225), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [27ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [27ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [44ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [47ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [47ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [84ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [87ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [89ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [89ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [220ms] refreshing cache from master, [220ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [224ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@259218191(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@6f703b29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611) -> (continuation of Deferred@1992745260 after retry RPC@1525975636), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25271bf -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611) -> (continuation of Deferred@1992745260 after retry RPC@1525975636))) after lookup
08:07:52.138 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:52.139 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.139 [DEBUG - kudu-nio-1] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.139 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.475 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:52.476 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)[L],ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:52.476 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=564), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [27ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [27ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [44ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [47ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [47ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [84ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [87ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [89ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [89ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [220ms] refreshing cache from master, [220ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [224ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [225ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [227ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [227ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [560ms] refreshing cache from master, [560ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [563ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@259218191(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@6f703b29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611) -> (continuation of Deferred@1992745260 after retry RPC@1525975636) -> (continuation of Deferred@2045012598 after retry RPC@487700555), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25271bf -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611) -> (continuation of Deferred@1992745260 after retry RPC@1525975636) -> (continuation of Deferred@2045012598 after retry RPC@487700555))) after lookup
08:07:52.477 [DEBUG - kudu-nio-1] (Connection.java:748) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] connecting to peer
08:07:52.478 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.478 [DEBUG - kudu-nio-0] (Connection.java:709) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] cleaning up while in state CONNECTING due to: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:52.478 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593
08:07:53.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.232200 17204 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e631edcb842e4e809e7bc9f4a63a388c)
08:07:53.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.232640 17204 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:53.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.235038 17204 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:53.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.239776 16668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:53.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.244082 16668 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:53.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.248862 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:53.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.249670 17093 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 9.
08:07:53.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.251060 16670 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c
08:07:53.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.251870 17204 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 10 FOLLOWER]: Leader pre-election won for term 11
08:07:53.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.252197 17204 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e631edcb842e4e809e7bc9f4a63a388c)
08:07:53.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.252561 17204 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 10 FOLLOWER]: Advancing to term 11
08:07:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.257455 17204 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.258931 17204 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 11 election: Requested vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:53.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.259858 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ec3940bcabac4a23852b5f27573cd9a1" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0"
08:07:53.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.260418 17093 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 9 FOLLOWER]: Advancing to term 11
08:07:53.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.262265 16668 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111)
08:07:53.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.267220 17093 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3940bcabac4a23852b5f27573cd9a1 in term 11.
08:07:53.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.268141 16670 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0; no voters: e631edcb842e4e809e7bc9f4a63a388c
08:07:53.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.268919 17204 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 11 FOLLOWER]: Leader election won for term 11
08:07:53.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.270650 17204 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 11 LEADER]: Becoming Leader. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Running, Role: LEADER
08:07:53.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.271574 17204 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:53.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.281932 13651 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 reported cstate change: term changed from 10 to 11, leader changed from e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66) to ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65). New cstate: current_term: 11 leader_uuid: "ec3940bcabac4a23852b5f27573cd9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: HEALTHY } } }
08:07:53.496 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:53.496 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:53.497 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1585), Traces: [0ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [2ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [2ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [24ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [25ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [27ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [27ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [44ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [45ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [47ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [47ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [84ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [87ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [89ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [89ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [220ms] refreshing cache from master, [220ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [224ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [225ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [227ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [227ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [560ms] refreshing cache from master, [560ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [563ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [564ms] sending RPC to server e631edcb842e4e809e7bc9f4a63a388c, [566ms] received response from server e631edcb842e4e809e7bc9f4a63a388c: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [566ms] delaying RPC due to: Network error: Failed to connect to peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593): Connection refused: /127.12.255.66:39593, [1580ms] refreshing cache from master, [1580ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1584ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@259218191(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@6f703b29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611) -> (continuation of Deferred@1992745260 after retry RPC@1525975636) -> (continuation of Deferred@2045012598 after retry RPC@487700555) -> (continuation of Deferred@2103524180 after retry RPC@1830946643), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@25271bf -> wakeup thread Time-limited test -> (continuation of Deferred@1320867980 after retry RPC@1601472365) -> (continuation of Deferred@1210949706 after retry RPC@226763611) -> (continuation of Deferred@1992745260 after retry RPC@1525975636) -> (continuation of Deferred@2045012598 after retry RPC@487700555) -> (continuation of Deferred@2103524180 after retry RPC@1830946643))) after lookup
08:07:53.497 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:53.498 [DEBUG - kudu-nio-1] (Connection.java:757) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] Successfully connected to peer
08:07:53.519 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x669ed352, L:/127.0.0.1:60756 - R:/127.12.255.65:38243] using TOKEN/null
08:07:53.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.529891 17093 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 11 FOLLOWER]: Refusing update from remote peer ec3940bcabac4a23852b5f27573cd9a1: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 38. (index mismatch)
08:07:53.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.531684 17213 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
08:07:53.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.532405 16668 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 -> Peer e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593): Couldn't send request to peer e631edcb842e4e809e7bc9f4a63a388c. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.66:39593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:53.595 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.66:39593
08:07:53.596 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:53.597397 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:53.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:39593
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42867
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:53.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.944711 17252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:53.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.945348 17252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:53.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.945962 17252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:53.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.979259 17252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:53.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:53.980237 17252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.66
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:54.015980 17252 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.66:39593
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.66
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42867
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.66
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/logs
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:54.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:54.017704 17252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:54.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:54.019546 17252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:55.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:54.038921 17261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:55.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:54.039304 17263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:55.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:54.040441 17252 server_base.cc:1034] running on GCE node
08:07:55.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:54.038902 17259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:07:55.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.323254 17252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:07:55.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.326058 17252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:07:55.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.327530 17252 hybrid_clock.cc:648] HybridClock initialized: now 1786522075327497 us; error 43 us; skew 500 ppm
08:07:55.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.328488 17252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:07:55.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.339041 17252 webserver.cc:460] Webserver started at http://127.12.255.66:42867/ using document root <none> and password file <none>
08:07:55.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.340081 17252 fs_manager.cc:359] Metadata directory not provided
08:07:55.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.340345 17252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:07:55.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.348603 17252 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
08:07:55.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.353200 17276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:07:55.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.354419 17252 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
08:07:55.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.354877 17252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:55.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:55.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:08 on dist-test-slave-ffrd"
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.357504 17252 fs_report.cc:383] FS layout report
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/wal
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/data
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:07:55.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:07:55.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.406304 17252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:07:55.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.408017 17252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:07:55.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.408437 17252 kserver.cc:163] Server-wide thread pool size limit: 3276
08:07:55.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.411178 17252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:07:55.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.418352 17286 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:07:55.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.426224 17252 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:07:55.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.426470 17252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
08:07:55.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.426789 17252 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:07:55.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.431725 17252 ts_tablet_manager.cc:610] Registered 1 tablets
08:07:55.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.431972 17252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
08:07:55.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.432535 17286 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap starting.
08:07:55.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.532446 17286 log.cc:824] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Log is configured to *not* fsync() on all Append() calls
08:07:55.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.646203 17286 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:07:55.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.647301 17286 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Bootstrap complete.
08:07:55.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.649068 17286 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent bootstrapping tablet: real 0.217s	user 0.168s	sys 0.032s
08:07:55.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.651211 17252 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.66:39593
08:07:55.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.651424 17461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.66:39593 every 8 connection(s)
08:07:55.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.654322 17252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-1/data/info.pb
08:07:55.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.656648 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 17252
08:07:55.657 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:55.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.676339 17286 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:55.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.681066 17286 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: e631edcb842e4e809e7bc9f4a63a388c, State: Initialized, Role: FOLLOWER
08:07:55.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.682372 17286 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:55.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.696246 17286 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c: Time spent starting tablet: real 0.047s	user 0.033s	sys 0.007s
08:07:55.731 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:07:55.732 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.12.255.65:38243
08:07:55.732 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:55.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.731173 17463 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:07:55.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.731711 17463 heartbeater.cc:464] Registering TS with master...
08:07:55.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.732702 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 16630
08:07:55.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.735306 17463 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:07:55.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.745344 17464 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:07:55.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.745783 17464 heartbeater.cc:464] Registering TS with master...
08:07:55.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.746690 17464 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:07:55.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.746953 13651 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:55.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.750743 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.66:45645
08:07:55.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.753533 13512 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:55.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.755884 17463 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:07:55.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.759337 17470 heartbeater.cc:347] Connected to a master server at 127.12.255.126:39387
08:07:55.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.759723 17470 heartbeater.cc:464] Registering TS with master...
08:07:55.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.760493 17470 heartbeater.cc:511] Master 127.12.255.126:39387 requested a full tablet report, sending...
08:07:55.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:55.763685 13390 ts_manager.cc:194] Re-registered known tserver with Master: e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593)
08:07:55.786 [DEBUG - kudu-nio-1] (Connection.java:239) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] handling channelInactive
08:07:55.787 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state READY due to: connection closed
08:07:55.788 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:07:55.789 [DEBUG - Time-limited test] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:55.790 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.790 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.790 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.796 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:55.797 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:55.798 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [8ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@14605058(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@3954ff1c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2023456f -> wakeup thread Time-limited test)) after lookup
08:07:55.798 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:55.799 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.799 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.799 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.816 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:55.816 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:55.816 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [8ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [10ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [11ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [11ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [27ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@14605058(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@3954ff1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2023456f -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662))) after lookup
08:07:55.817 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:55.818 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.818 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.818 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.855 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:55.855 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:55.856 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [8ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [10ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [11ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [11ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [27ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [29ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [30ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [30ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [67ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@14605058(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@3954ff1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2023456f -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865))) after lookup
08:07:55.857 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:55.858 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.858 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.858 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.955 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:55.956 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:55.956 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=168), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [8ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [10ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [11ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [11ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [27ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [29ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [30ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [30ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [67ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [68ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [70ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [70ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [167ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@14605058(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@3954ff1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2023456f -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290))) after lookup
08:07:55.957 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:55.958 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.958 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:55.958 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:56.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:56.076598 17159 debug-util.cc:398] Leaking SignalData structure 0x7b08000d58e0 after lost signal to thread 16936
08:07:56.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:56.077618 17159 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7fa0 after lost signal to thread 17167
08:07:56.176 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:56.177 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:56.177 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=389), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [8ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [10ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [11ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [11ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [27ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [29ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [30ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [30ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [67ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [68ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [70ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [70ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [167ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [169ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [170ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [170ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [384ms] refreshing cache from master, [384ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [388ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@14605058(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@3954ff1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290) -> (continuation of Deferred@1666797861 after retry RPC@1536186641), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2023456f -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290) -> (continuation of Deferred@1666797861 after retry RPC@1536186641))) after lookup
08:07:56.178 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:56.179 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:56.179 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:56.179 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:56.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:56.963726 17479 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:07:56.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:56.964166 17479 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:56.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:56.966450 17479 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67:43597), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:56.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:56.970276 17282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:56.977 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:56.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:56.976679 17282 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:56.978 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)[L],ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)]}]
08:07:56.978 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1190), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [8ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [10ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [11ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [11ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [27ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [29ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [30ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [30ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [67ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [68ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [70ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [70ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [167ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [169ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [170ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [170ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [384ms] refreshing cache from master, [384ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [388ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [390ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [391ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [391ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [1184ms] refreshing cache from master, [1184ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1189ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@14605058(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@3954ff1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290) -> (continuation of Deferred@1666797861 after retry RPC@1536186641) -> (continuation of Deferred@692348915 after retry RPC@1889629676), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2023456f -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290) -> (continuation of Deferred@1666797861 after retry RPC@1536186641) -> (continuation of Deferred@692348915 after retry RPC@1889629676))) after lookup
08:07:56.980 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] connecting to peer
08:07:56.982 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:56.982 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] cleaning up while in state CONNECTING due to: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:56.982 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243) for tablet ccffce57a953436db65493227b7dd42b: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243
08:07:56.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:56.984148 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "e631edcb842e4e809e7bc9f4a63a388c" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "ff4960fadb0549a59a1e29815dde2bb0" is_pre_election: true
08:07:56.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:56.985853 17280 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c; no voters: ec3940bcabac4a23852b5f27573cd9a1, ff4960fadb0549a59a1e29815dde2bb0
08:07:56.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:56.986918 17479 raft_consensus.cc:2726] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
08:07:57.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.227304 17489 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:57.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.227777 17489 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:57.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.230530 17489 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:57.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:57.236450 16971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:57.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:57.241837 16971 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:57.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.246798 17392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c" is_pre_election: true
08:07:57.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.247802 17392 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 10.
08:07:57.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.249648 16968 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:57.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.250578 17489 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 11 FOLLOWER]: Leader pre-election won for term 12
08:07:57.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.250995 17489 raft_consensus.cc:482] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ec3940bcabac4a23852b5f27573cd9a1)
08:07:57.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.251354 17489 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 11 FOLLOWER]: Advancing to term 12
08:07:57.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.256676 17489 raft_consensus.cc:504] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:57.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.258332 17489 leader_election.cc:288] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 12 election: Requested vote from peers e631edcb842e4e809e7bc9f4a63a388c (127.12.255.66:39593), ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:07:57.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.259402 17392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccffce57a953436db65493227b7dd42b" candidate_uuid: "ff4960fadb0549a59a1e29815dde2bb0" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "e631edcb842e4e809e7bc9f4a63a388c"
08:07:57.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.260075 17392 raft_consensus.cc:3037] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 10 FOLLOWER]: Advancing to term 12
08:07:57.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:57.262106 16971 leader_election.cc:334] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111)
08:07:57.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.270900 17392 raft_consensus.cc:2454] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff4960fadb0549a59a1e29815dde2bb0 in term 12.
08:07:57.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.272066 16968 leader_election.cc:302] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e631edcb842e4e809e7bc9f4a63a388c, ff4960fadb0549a59a1e29815dde2bb0; no voters: ec3940bcabac4a23852b5f27573cd9a1
08:07:57.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.272933 17489 raft_consensus.cc:2781] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 12 FOLLOWER]: Leader election won for term 12
08:07:57.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.275125 17489 raft_consensus.cc:686] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [term 12 LEADER]: Becoming Leader. State: Replica: ff4960fadb0549a59a1e29815dde2bb0, State: Running, Role: LEADER
08:07:57.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.276763 17489 consensus_queue.cc:237] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:07:57.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.290402 13651 catalog_manager.cc:5495] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 reported cstate change: term changed from 11 to 12, leader changed from ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65) to ff4960fadb0549a59a1e29815dde2bb0 (127.12.255.67). New cstate: current_term: 12 leader_uuid: "ff4960fadb0549a59a1e29815dde2bb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } health_report { overall_health: UNKNOWN } } }
08:07:57.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:57.854002 16971 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 -> Peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Couldn't send request to peer ec3940bcabac4a23852b5f27573cd9a1. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:07:57.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.856411 17392 raft_consensus.cc:1261] T ccffce57a953436db65493227b7dd42b P e631edcb842e4e809e7bc9f4a63a388c [term 12 FOLLOWER]: Refusing update from remote peer ff4960fadb0549a59a1e29815dde2bb0: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
08:07:57.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:57.857985 17514 consensus_queue.cc:1035] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
08:07:59.376 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet ccffce57a953436db65493227b7dd42b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695' with partition [<start>, <end>)
08:07:59.376 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=ccffce57a953436db65493227b7dd42b@[e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593),ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243),ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)[L]]}]
08:07:59.377 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=ccffce57a953436db65493227b7dd42b, attempt=14, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=3588), Traces: [0ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [2ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [2ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [8ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [10ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [11ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [11ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [27ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [29ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [30ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [30ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [67ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [68ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [70ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [70ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [164ms] refreshing cache from master, [164ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [167ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [169ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [170ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [170ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [384ms] refreshing cache from master, [384ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [388ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [390ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [391ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [391ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [1184ms] refreshing cache from master, [1184ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [1189ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, [1191ms] sending RPC to server ec3940bcabac4a23852b5f27573cd9a1, [1194ms] received response from server ec3940bcabac4a23852b5f27573cd9a1: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [1194ms] delaying RPC due to: Network error: Failed to connect to peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243): Connection refused: /127.12.255.65:38243, [3584ms] refreshing cache from master, [3584ms] Sub RPC GetTableLocations: sending RPC to server master-127.12.255.124:36715, [3587ms] Sub RPC GetTableLocations: received response from server master-127.12.255.124:36715: OK, deferred=Deferred@14605058(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/1991069968@3954ff1c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290) -> (continuation of Deferred@1666797861 after retry RPC@1536186641) -> (continuation of Deferred@692348915 after retry RPC@1889629676) -> (continuation of Deferred@798549228 after retry RPC@1424878310), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2023456f -> wakeup thread Time-limited test -> (continuation of Deferred@1092497524 after retry RPC@2131829662) -> (continuation of Deferred@1681533979 after retry RPC@306310865) -> (continuation of Deferred@38731389 after retry RPC@1499215290) -> (continuation of Deferred@1666797861 after retry RPC@1536186641) -> (continuation of Deferred@692348915 after retry RPC@1889629676) -> (continuation of Deferred@798549228 after retry RPC@1424878310))) after lookup
08:07:59.378 [DEBUG - kudu-nio-1] (Connection.java:748) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] connecting to peer
08:07:59.379 [DEBUG - kudu-nio-1] (Connection.java:757) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] Successfully connected to peer
08:07:59.398 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x64dcc2eb, L:/127.0.0.1:35534 - R:/127.12.255.67:43597] using TOKEN/null
08:07:59.448 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.12.255.65:38243
08:07:59.448 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:59.450821 13309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:07:59.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:59.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:07:59.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.807760 17566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
08:07:59.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.808406 17566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:07:59.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.809104 17566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
08:07:59.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.842060 17566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
08:07:59.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.843071 17566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.255.65
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:59.884786 17566 tablet_server_runner.cc:78] Tablet server non-default flags:
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.12.255.84:34905
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.12.255.65:38243
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.12.255.65
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46817
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.12.255.126:39387,127.12.255.125:46135,127.12.255.124:36715
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:07:59.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.12.255.65
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/logs
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 12 Aug 2026 06:37:03 UTC on ccycloud.quasar-wifqrz.root.comops.site
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4581996
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
08:07:59.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:59.886435 17566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:07:59.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:07:59.888324 17566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
08:07:59.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.909533 17581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:08:00.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:08:00.368063 16971 consensus_peers.cc:487] T ccffce57a953436db65493227b7dd42b P ff4960fadb0549a59a1e29815dde2bb0 -> Peer ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243): Couldn't send request to peer ec3940bcabac4a23852b5f27573cd9a1. Status: Network error: Client connection negotiation failed: client connection to 127.12.255.65:38243: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:08:01.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.910035 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:08:01.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:07:59.909611 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:08:01.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:08:01.208738 17586 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1296 milliseconds
08:08:01.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.208881 17566 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
08:08:01.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.210247 17566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:08:01.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.213146 17566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:08:01.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.214602 17566 hybrid_clock.cc:648] HybridClock initialized: now 1786522081214557 us; error 64 us; skew 500 ppm
08:08:01.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.215557 17566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:08:01.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.225075 17566 webserver.cc:460] Webserver started at http://127.12.255.65:46817/ using document root <none> and password file <none>
08:08:01.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.226027 17566 fs_manager.cc:359] Metadata directory not provided
08:08:01.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.226225 17566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:08:01.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.234337 17566 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.002s
08:08:01.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.238936 17608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:08:01.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.240151 17566 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
08:08:01.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.240463 17566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data,/tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ec3940bcabac4a23852b5f27573cd9a1"
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:07:05 on dist-test-slave-ffrd"
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.242964 17566 fs_report.cc:383] FS layout report
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/wal
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/data
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:08:01.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:08:01.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.287547 17566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
08:08:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.289299 17566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:08:01.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.289834 17566 kserver.cc:163] Server-wide thread pool size limit: 3276
08:08:01.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.292438 17566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
08:08:01.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.298439 17617 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
08:08:01.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.306166 17566 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:08:01.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.306494 17566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
08:08:01.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.306916 17566 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
08:08:01.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.312310 17566 ts_tablet_manager.cc:610] Registered 1 tablets
08:08:01.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.312529 17566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.000s	sys 0.003s
08:08:01.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.313020 17617 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap starting.
08:08:01.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.387928 17617 log.cc:824] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Log is configured to *not* fsync() on all Append() calls
08:08:01.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.503949 17566 rpc_server.cc:307] RPC server started. Bound to: 127.12.255.65:38243
08:08:01.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.504135 17805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.255.65:38243 every 8 connection(s)
08:08:01.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.506767 17566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM4sYl4/test-tmp/mini-kudu-cluster8528732591651631563/ts-0/data/info.pb
08:08:01.510 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:08:01.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.509931 13309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu as pid 17566
08:08:01.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.538302 17617 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:08:01.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.539423 17617 tablet_bootstrap.cc:492] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Bootstrap complete.
08:08:01.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.541332 17617 ts_tablet_manager.cc:1397] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent bootstrapping tablet: real 0.229s	user 0.167s	sys 0.047s
08:08:01.574 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:08:01.593 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522030695, tablet=ccffce57a953436db65493227b7dd42b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:08:01.610 [DEBUG - main] (Connection.java:575) [peer master-127.12.255.126:39387(127.12.255.126:39387)] disconnecting while in state READY
08:08:01.615 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.12.255.126:39387(127.12.255.126:39387)] handling channelInactive
08:08:01.615 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.12.255.126:39387(127.12.255.126:39387)] cleaning up while in state READY due to: connection closed
08:08:01.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.615509 17809 heartbeater.cc:347] Connected to a master server at 127.12.255.125:46135
08:08:01.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.616138 17809 heartbeater.cc:464] Registering TS with master...
08:08:01.616 [DEBUG - main] (Connection.java:575) [peer e631edcb842e4e809e7bc9f4a63a388c(127.12.255.66:39593)] disconnecting while in state TERMINATED
08:08:01.617 [DEBUG - main] (Connection.java:575) [peer master-127.12.255.125:46135(127.12.255.125:46135)] disconnecting while in state READY
08:08:01.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.617673 17809 heartbeater.cc:511] Master 127.12.255.125:46135 requested a full tablet report, sending...
08:08:01.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.620463 17806 heartbeater.cc:347] Connected to a master server at 127.12.255.124:36715
08:08:01.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.620796 17806 heartbeater.cc:464] Registering TS with master...
08:08:01.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.621662 17806 heartbeater.cc:511] Master 127.12.255.124:36715 requested a full tablet report, sending...
08:08:01.622 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.12.255.125:46135(127.12.255.125:46135)] handling channelInactive
08:08:01.622 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.12.255.125:46135(127.12.255.125:46135)] cleaning up while in state READY due to: connection closed
08:08:01.623 [DEBUG - main] (Connection.java:575) [peer ec3940bcabac4a23852b5f27573cd9a1(127.12.255.65:38243)] disconnecting while in state TERMINATED
08:08:01.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.625371 13651 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:08:01.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.625502 13512 ts_manager.cc:194] Re-registered known tserver with Master: ec3940bcabac4a23852b5f27573cd9a1 (127.12.255.65:38243)
08:08:01.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.629673 13651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.255.65:59473
08:08:01.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.624575 17617 raft_consensus.cc:348] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:08:01.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.630187 17617 raft_consensus.cc:729] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3940bcabac4a23852b5f27573cd9a1, State: Initialized, Role: FOLLOWER
08:08:01.631 [DEBUG - main] (Connection.java:575) [peer master-127.12.255.124:36715(127.12.255.124:36715)] disconnecting while in state READY
08:08:01.632 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.12.255.124:36715(127.12.255.124:36715)] handling channelInactive
08:08:01.633 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.12.255.124:36715(127.12.255.124:36715)] cleaning up while in state READY due to: connection closed
08:08:01.634 [DEBUG - main] (Connection.java:575) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] disconnecting while in state READY
08:08:01.636 [DEBUG - kudu-nio-1] (Connection.java:239) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] handling channelInactive
08:08:01.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.636519 17617 consensus_queue.cc:260] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4960fadb0549a59a1e29815dde2bb0" member_type: VOTER last_known_addr { host: "127.12.255.67" port: 43597 } } peers { permanent_uuid: "e631edcb842e4e809e7bc9f4a63a388c" member_type: VOTER last_known_addr { host: "127.12.255.66" port: 39593 } } peers { permanent_uuid: "ec3940bcabac4a23852b5f27573cd9a1" member_type: VOTER last_known_addr { host: "127.12.255.65" port: 38243 } }
08:08:01.640 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ff4960fadb0549a59a1e29815dde2bb0(127.12.255.67:43597)] cleaning up while in state READY due to: connection closed
08:08:01.641 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
08:08:01.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.652587 17806 heartbeater.cc:503] Master 127.12.255.124:36715 was elected leader, sending a full tablet report...
08:08:01.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.652740 17617 ts_tablet_manager.cc:1428] T ccffce57a953436db65493227b7dd42b P ec3940bcabac4a23852b5f27573cd9a1: Time spent starting tablet: real 0.111s	user 0.031s	sys 0.006s
08:08:01.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.668247 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 17566
08:08:01.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:08:01.715476 17827 negotiation.cc:337] Failed RPC negotiation. Trace:
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.584835 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.255.65:33901 (local address 127.12.255.126:39387)
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.673565 (+ 88730us) server_negotiation.cc:206] Beginning negotiation
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.673599 (+    34us) server_negotiation.cc:399] Waiting for connection header
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.673670 (+    71us) server_negotiation.cc:407] Connection header received
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.673878 (+   208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.673902 (+    24us) server_negotiation.cc:461] Received NEGOTIATE request from client
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.674169 (+   267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.674396 (+   227us) server_negotiation.cc:226] Negotiated authn=SASL
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:08:01.713849 (+ 39453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.255.65:33901: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":88208,"thread_start_us":90270,"threads_started":1}
08:08:01.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:08:01.716360 13487 connection.cc:535] server connection from 127.12.255.65:38429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:08:01.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.720156 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 17252
08:08:01.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.785815 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 16935
08:08:01.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.831235 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 13328
08:08:01.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.877043 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 13449
08:08:01.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:08:01.920147 13309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM4sYl4/build/tsan/bin/kudu with pid 13572
08:08:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:08:03Z chronyd exiting

Time: 69.425

OK (1 test)

