JUnit version 4.13.2
.16:15:30.470 [INFO - main] (RandomUtils.java:49) Using random seed: 1786464930464
16:15:30.515 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
16:15:30.574 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
16:15:30.589 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskQnSfSC/test-tmp
16:15:30.591 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
16:15:30.595 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskQnSfSC/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
16:15:30.865 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

16:15:31.034 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
16:15:31.035 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

16:15:31.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.039054 22653 mini_oidc.cc:152] Starting JWKS server
16:15:31.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.043098 22653 webserver.cc:460] Webserver started at http://127.0.0.1:35555/ using document root <none> and password file <none>
16:15:31.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.043653 22653 mini_oidc.cc:189] Starting OIDC Discovery server
16:15:31.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.045881 22653 webserver.cc:460] Webserver started at http://127.0.0.1:41153/ using document root <none> and password file <none>
16:15:31.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T16:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
16:15:31.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T16:15:31Z Disabled control of system clock
16:15:31.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.110767 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:31.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:31.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/wal
16:15:31.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data
16:15:31.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:31.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/logs
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data/info.pb
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:31.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:31.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:31.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:31.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:31.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:31.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:31.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:31.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.126:42753
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.126
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:31.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:31.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:31.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:15:31.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:31.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:15:31.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35555/jwks.json with env {}
16:15:31.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.459444 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:31.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.460260 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:31.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.461030 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:31.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.494645 22666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:15:31.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.495042 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:31.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.495405 22666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:15:31.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.495749 22666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:15:31.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.496140 22666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:15:31.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.496441 22666 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35555/jwks.json
16:15:31.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.533896 22666 master_runner.cc:386] Master server non-default flags:
16:15:31.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:31.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:31.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:31.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:31.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data
16:15:31.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/wal
16:15:31.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:15:31.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:31.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:31.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:31.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.126:42753
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35555/jwks.json
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:31.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data/info.pb
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.126
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/logs
16:15:31.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:31.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:31.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.535907 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:31.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.538980 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:31.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.561697 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:32.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.562112 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:32.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:31.563297 22666 server_base.cc:1034] running on GCE node
16:15:32.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:31.563374 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:32.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.813170 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:32.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.816700 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:32.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.818266 22666 hybrid_clock.cc:648] HybridClock initialized: now 1786464932818214 us; error 135 us; skew 500 ppm
16:15:32.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.819129 22666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:32.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.833536 22666 webserver.cc:460] Webserver started at http://127.22.31.126:37385/ using document root <none> and password file <none>
16:15:32.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.834692 22666 fs_manager.cc:359] Metadata directory not provided
16:15:32.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.834896 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:32.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.835312 22666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:15:32.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.840729 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data/instance:
16:15:32.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e60025b429fc4965b452cf68a21b35de"
16:15:32.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:32 on dist-test-slave-9jjr"
16:15:32.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.842216 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/wal/instance:
16:15:32.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e60025b429fc4965b452cf68a21b35de"
16:15:32.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:32 on dist-test-slave-9jjr"
16:15:32.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.850901 22666 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.006s	sys 0.001s
16:15:32.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.858300 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:32.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.859901 22666 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.001s
16:15:32.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.860257 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/wal
16:15:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e60025b429fc4965b452cf68a21b35de"
16:15:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:32 on dist-test-slave-9jjr"
16:15:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.860713 22666 fs_report.cc:383] FS layout report
16:15:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:32.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/wal
16:15:32.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/wal
16:15:32.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data/data
16:15:32.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:32.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:32.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.905261 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:32.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.907322 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:32.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.907829 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:32.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.992871 22666 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.126:42753
16:15:32.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.992942 22742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.126:42753 every 8 connection(s)
16:15:32.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:32.995810 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/data/info.pb
16:15:33.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.002483 22745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:15:33.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.004060 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 22666
16:15:33.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.004630 22653 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-0/wal/instance
16:15:33.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.008802 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:33.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:33.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/wal
16:15:33.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data
16:15:33.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/logs
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data/info.pb
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:33.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:33.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:15:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:15:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.125:39741
16:15:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.125
16:15:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:33.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:33.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:33.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:15:33.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:33.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:15:33.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35555/jwks.json with env {}
16:15:33.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.016971 22745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } has no permanent_uuid. Determining permanent_uuid...
16:15:33.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.052325 22745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } has no permanent_uuid. Determining permanent_uuid...
16:15:33.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.054787 22686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:33.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.059644 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.125:39741: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:33.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.109417 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } attempt: 1
16:15:33.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.113984 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.125:39741: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:33.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.182555 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } attempt: 2
16:15:33.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.186578 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.125:39741: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:33.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.278167 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } attempt: 3
16:15:33.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.283741 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.125:39741: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.375330 22747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.376055 22747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.376573 22747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:33.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.410921 22747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:15:33.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.411258 22747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:33.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.411463 22747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:15:33.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.411654 22747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:15:33.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.411839 22747 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:15:33.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.411990 22747 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35555/jwks.json
16:15:33.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.427238 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } attempt: 4
16:15:33.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.431782 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.125:39741: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:33.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.451653 22747 master_runner.cc:386] Master server non-default flags:
16:15:33.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:33.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:33.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:33.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/wal
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.125:39741
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35555/jwks.json
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data/info.pb
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.125
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:33.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/logs
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:15:33.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.453012 22747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.455346 22747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:33.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.478682 22759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:33.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.731299 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } attempt: 5
16:15:33.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.736702 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.125:39741: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:34.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.284471 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } attempt: 6
16:15:34.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:34.292690 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.125:39741: Network error: Client connection negotiation failed: client connection to 127.22.31.125:39741: connect: Connection refused (error 111)
16:15:34.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.478950 22761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:34.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:33.478973 22758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:34.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:33.480973 22747 server_base.cc:1034] running on GCE node
16:15:34.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.764904 22747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:34.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.768802 22747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:34.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.770324 22747 hybrid_clock.cc:648] HybridClock initialized: now 1786464934770269 us; error 73 us; skew 500 ppm
16:15:34.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.772759 22747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:34.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.781425 22747 webserver.cc:460] Webserver started at http://127.22.31.125:42647/ using document root <none> and password file <none>
16:15:34.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.783095 22747 fs_manager.cc:359] Metadata directory not provided
16:15:34.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.783432 22747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:34.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.784070 22747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:15:34.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.790196 22747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data/instance:
16:15:34.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6d92f4e01a544dd7954ef13f03150ee9"
16:15:34.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:34 on dist-test-slave-9jjr"
16:15:34.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.791548 22747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/wal/instance:
16:15:34.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6d92f4e01a544dd7954ef13f03150ee9"
16:15:34.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:34 on dist-test-slave-9jjr"
16:15:34.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.800942 22747 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.004s
16:15:34.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.807966 22771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:34.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.809384 22747 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
16:15:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.809718 22747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/wal
16:15:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6d92f4e01a544dd7954ef13f03150ee9"
16:15:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:34 on dist-test-slave-9jjr"
16:15:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.810074 22747 fs_report.cc:383] FS layout report
16:15:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/wal
16:15:34.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/wal
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data/data
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:34.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:34.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:34.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:34.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:34.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.868742 22747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:34.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.870832 22747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:34.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.871316 22747 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:34.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.966241 22747 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.125:39741
16:15:34.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.966320 22827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.125:39741 every 8 connection(s)
16:15:34.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.969290 22747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/data/info.pb
16:15:34.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.976018 22828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:15:34.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.977952 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 22747
16:15:34.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.978708 22653 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-1/wal/instance
16:15:34.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.981156 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:34.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:34.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/wal
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/logs
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data/info.pb
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:34.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:15:34.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.124:46519
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.124
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:34.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:15:34.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35555/jwks.json with env {}
16:15:34.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:34.992581 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } has no permanent_uuid. Determining permanent_uuid...
16:15:35.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.033471 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } has no permanent_uuid. Determining permanent_uuid...
16:15:35.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.054571 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } has no permanent_uuid. Determining permanent_uuid...
16:15:35.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.057621 22773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.062918 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.112617 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 1
16:15:35.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.117484 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.185987 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 2
16:15:35.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.190912 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.282421 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 3
16:15:35.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.286850 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.353408 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } attempt: 7
16:15:35.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.368916 22745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } has no permanent_uuid. Determining permanent_uuid...
16:15:35.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.375579 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.380009 22830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:35.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.380924 22830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:35.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.381584 22830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:35.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.414839 22830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:15:35.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.415165 22830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:35.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.415400 22830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:15:35.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.415583 22830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:15:35.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.415757 22830 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:15:35.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.415899 22830 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35555/jwks.json
16:15:35.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.430337 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 4
16:15:35.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.434087 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 1
16:15:35.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.435490 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.438686 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.457931 22830 master_runner.cc:386] Master server non-default flags:
16:15:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/wal
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.124:46519
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35555/jwks.json
16:15:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data/info.pb
16:15:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.124
16:15:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/logs
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.459666 22830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.462018 22830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:35.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.482880 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:35.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.520128 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 2
16:15:35.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.525146 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.610675 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 3
16:15:35.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.614822 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.735033 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 5
16:15:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.742333 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:35.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.755575 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 4
16:15:35.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.762218 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:36.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.045780 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 5
16:15:36.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:36.049801 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:36.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.290186 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 6
16:15:36.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:36.298452 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:36.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.602494 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 6
16:15:36.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:36.612375 22745 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.31.124:46519: Network error: Client connection negotiation failed: client connection to 127.22.31.124:46519: connect: Connection refused (error 111)
16:15:36.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.483083 22845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:36.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:35.488801 22830 server_base.cc:1034] running on GCE node
16:15:36.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:35.483698 22842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:36.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.772302 22830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:36.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.775096 22830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:36.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.776564 22830 hybrid_clock.cc:648] HybridClock initialized: now 1786464936776535 us; error 68 us; skew 500 ppm
16:15:36.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.777638 22830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:36.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.786650 22830 webserver.cc:460] Webserver started at http://127.22.31.124:34015/ using document root <none> and password file <none>
16:15:36.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.788000 22830 fs_manager.cc:359] Metadata directory not provided
16:15:36.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.788276 22830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:36.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.788830 22830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:15:36.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.794195 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data/instance:
16:15:36.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "490a6a8e3d3b4cadbae2114293980059"
16:15:36.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:36 on dist-test-slave-9jjr"
16:15:36.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.795502 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/wal/instance:
16:15:36.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "490a6a8e3d3b4cadbae2114293980059"
16:15:36.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:36 on dist-test-slave-9jjr"
16:15:36.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.803679 22830 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.004s
16:15:36.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.809818 22860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:36.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.811292 22830 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.811702 22830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/wal
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "490a6a8e3d3b4cadbae2114293980059"
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:36 on dist-test-slave-9jjr"
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.812209 22830 fs_report.cc:383] FS layout report
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/wal
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/wal
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data/data
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:36.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:36.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:36.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.848518 22830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:36.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.850590 22830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:36.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.851075 22830 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:36.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.952360 22830 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.124:46519
16:15:36.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.952488 22920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.124:46519 every 8 connection(s)
16:15:36.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.955614 22830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/data/info.pb
16:15:36.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.956593 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 22830
16:15:36.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.957072 22653 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/master-2/wal/instance
16:15:36.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.965067 22921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:15:36.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:36.992558 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } has no permanent_uuid. Determining permanent_uuid...
16:15:37.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.034590 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } has no permanent_uuid. Determining permanent_uuid...
16:15:37.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.048400 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } has no permanent_uuid. Determining permanent_uuid...
16:15:37.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.072474 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059: Bootstrap starting.
16:15:37.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.080921 22921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059: Neither blocks nor log segments found. Creating new log.
16:15:37.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.084882 22921 log.cc:824] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059: Log is configured to *not* fsync() on all Append() calls
16:15:37.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.092340 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059: No bootstrap required, opened a new log
16:15:37.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.120523 22921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.121219 22921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:15:37.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.121528 22921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490a6a8e3d3b4cadbae2114293980059, State: Initialized, Role: FOLLOWER
16:15:37.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.122566 22921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.126675 22930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.127524 22930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: This master's current role is: FOLLOWER
16:15:37.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.128887 22921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: configured and running, proceeding with master startup.
16:15:37.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:37.166827 22941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
16:15:37.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:37.167253 22941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
16:15:37.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.161796 22830 master_runner.cc:186] Error getting master registration for 127.22.31.126:42753: OK, instance_id { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" instance_seqno: 1786464932968711 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:15:37.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.197942 22830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:15:37.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:37.357518 22867 tablet.cc:2369] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:15:37.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.359130 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 7
16:15:37.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.387852 22828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: Bootstrap starting.
16:15:37.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.398365 22828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: Neither blocks nor log segments found. Creating new log.
16:15:37.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.401621 22828 log.cc:824] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: Log is configured to *not* fsync() on all Append() calls
16:15:37.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.408743 22828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: No bootstrap required, opened a new log
16:15:37.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.444487 22828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.445624 22828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:15:37.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.445957 22828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d92f4e01a544dd7954ef13f03150ee9, State: Initialized, Role: FOLLOWER
16:15:37.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.447162 22828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.450474 22947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.451642 22947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [sys.catalog]: This master's current role is: FOLLOWER
16:15:37.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.452024 22828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [sys.catalog]: configured and running, proceeding with master startup.
16:15:37.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.498654 22930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:15:37.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.499550 22930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.508769 22930 leader_election.cc:288] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e60025b429fc4965b452cf68a21b35de (127.22.31.126:42753), 6d92f4e01a544dd7954ef13f03150ee9 (127.22.31.125:39741)
16:15:37.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.507352 22801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "490a6a8e3d3b4cadbae2114293980059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92f4e01a544dd7954ef13f03150ee9" is_pre_election: true
16:15:37.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.510537 22801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 490a6a8e3d3b4cadbae2114293980059 in term 0.
16:15:37.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.513442 22863 leader_election.cc:302] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490a6a8e3d3b4cadbae2114293980059, 6d92f4e01a544dd7954ef13f03150ee9; no voters: 
16:15:37.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.514214 22930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Leader pre-election won for term 1
16:15:37.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.514701 22930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:15:37.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.515053 22930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 0 FOLLOWER]: Advancing to term 1
16:15:37.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:37.523986 22961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
16:15:37.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.525002 22930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:37.525867 22961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
16:15:37.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.527908 22930 leader_election.cc:288] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [CANDIDATE]: Term 1 election: Requested vote from peers e60025b429fc4965b452cf68a21b35de (127.22.31.126:42753), 6d92f4e01a544dd7954ef13f03150ee9 (127.22.31.125:39741)
16:15:37.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.529543 22801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "490a6a8e3d3b4cadbae2114293980059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92f4e01a544dd7954ef13f03150ee9"
16:15:37.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.530319 22801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [term 0 FOLLOWER]: Advancing to term 1
16:15:37.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.543982 22801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 490a6a8e3d3b4cadbae2114293980059 in term 1.
16:15:37.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.495537 22747 master_runner.cc:186] Error getting master registration for 127.22.31.126:42753: OK, instance_id { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" instance_seqno: 1786464932968711 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:15:37.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.557559 22747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:15:37.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.563154 22863 leader_election.cc:302] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490a6a8e3d3b4cadbae2114293980059, 6d92f4e01a544dd7954ef13f03150ee9; no voters: 
16:15:37.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.564182 22930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 1 FOLLOWER]: Leader election won for term 1
16:15:37.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.575924 22930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [term 1 LEADER]: Becoming Leader. State: Replica: 490a6a8e3d3b4cadbae2114293980059, State: Running, Role: LEADER
16:15:37.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.578207 22930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.609381 22963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490a6a8e3d3b4cadbae2114293980059. Latest consensus state: current_term: 1 leader_uuid: "490a6a8e3d3b4cadbae2114293980059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.610342 22963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: This master's current role is: LEADER
16:15:37.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.618822 22965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
16:15:37.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.629855 22965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
16:15:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:37.634239 22777 tablet.cc:2369] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:15:37.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.645867 22745 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } attempt: 7
16:15:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.681149 22801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [term 1 FOLLOWER]: Refusing update from remote peer 490a6a8e3d3b4cadbae2114293980059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:15:37.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.687034 22930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
16:15:37.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.701830 22745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: Bootstrap starting.
16:15:37.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.737700 22745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: Neither blocks nor log segments found. Creating new log.
16:15:37.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.752271 22745 log.cc:824] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: Log is configured to *not* fsync() on all Append() calls
16:15:37.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.769495 22947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490a6a8e3d3b4cadbae2114293980059. Latest consensus state: current_term: 1 leader_uuid: "490a6a8e3d3b4cadbae2114293980059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.772292 22947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [sys.catalog]: This master's current role is: FOLLOWER
16:15:37.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.774161 22968 mvcc.cc:205] Tried to move back new op lower bound from 7317360384710889472 to 7317360384337793024. Current Snapshot: MvccSnapshot[applied={T|T < 7317360384710889472}]
16:15:37.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.777702 22745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: No bootstrap required, opened a new log
16:15:37.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.794498 22963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "490a6a8e3d3b4cadbae2114293980059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.795408 22963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: This master's current role is: LEADER
16:15:37.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.809044 22947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "490a6a8e3d3b4cadbae2114293980059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.810107 22947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9 [sys.catalog]: This master's current role is: FOLLOWER
16:15:37.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.819041 22965 catalog_manager.cc:1347] Generated new cluster ID: 6e7d4b08868d47c88f9468a8da4959e1
16:15:37.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.819396 22965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
16:15:37.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.840373 22745 raft_consensus.cc:348] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.841884 22745 raft_consensus.cc:374] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:15:37.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.842535 22745 raft_consensus.cc:729] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60025b429fc4965b452cf68a21b35de, State: Initialized, Role: FOLLOWER
16:15:37.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.843870 22745 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } }
16:15:37.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.849799 22745 sys_catalog.cc:564] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [sys.catalog]: configured and running, proceeding with master startup.
16:15:37.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.850323 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "490a6a8e3d3b4cadbae2114293980059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60025b429fc4965b452cf68a21b35de"
16:15:37.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.851403 22712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "490a6a8e3d3b4cadbae2114293980059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60025b429fc4965b452cf68a21b35de" is_pre_election: true
16:15:37.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.852717 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [term 0 FOLLOWER]: Advancing to term 1
16:15:37.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.853132 22975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.870323 22975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [sys.catalog]: This master's current role is: FOLLOWER
16:15:37.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.879916 22710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [term 1 FOLLOWER]: Refusing update from remote peer 490a6a8e3d3b4cadbae2114293980059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:15:37.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.889295 22930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
16:15:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.907214 22965 catalog_manager.cc:1370] Generated new certificate authority record
16:15:37.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.910321 22965 catalog_manager.cc:1504] Loading token signing keys...
16:15:37.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:37.921195 22690 tablet.cc:2369] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:15:37.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.952868 22965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059: Generated new TSK 0
16:15:37.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.954195 22965 catalog_manager.cc:1514] Initializing in-progress tserver states...
16:15:37.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.996973 22963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "490a6a8e3d3b4cadbae2114293980059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:37.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:37.997815 22963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 490a6a8e3d3b4cadbae2114293980059 [sys.catalog]: This master's current role is: LEADER
16:15:38.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.015316 22975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490a6a8e3d3b4cadbae2114293980059. Latest consensus state: current_term: 1 leader_uuid: "490a6a8e3d3b4cadbae2114293980059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:38.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.016542 22975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [sys.catalog]: This master's current role is: FOLLOWER
16:15:38.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.018831 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "490a6a8e3d3b4cadbae2114293980059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60025b429fc4965b452cf68a21b35de" member_type: VOTER last_known_addr { host: "127.22.31.126" port: 42753 } } peers { permanent_uuid: "6d92f4e01a544dd7954ef13f03150ee9" member_type: VOTER last_known_addr { host: "127.22.31.125" port: 39741 } } peers { permanent_uuid: "490a6a8e3d3b4cadbae2114293980059" member_type: VOTER last_known_addr { host: "127.22.31.124" port: 46519 } } }
16:15:38.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.020210 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de [sys.catalog]: This master's current role is: FOLLOWER
16:15:38.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.022238 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:38.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:38.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:38.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:0
16:15:38.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:15:38.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:15:38.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:38.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:38.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:38.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:38.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:38.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:15:38.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.147298 22996 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
16:15:38.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.534392 22961 catalog_manager.cc:1259] Loaded cluster ID: 6e7d4b08868d47c88f9468a8da4959e1
16:15:38.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.535276 22961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: loading cluster ID for follower catalog manager: success
16:15:38.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.547871 22961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: acquiring CA information for follower catalog manager: success
16:15:38.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.556910 22961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d92f4e01a544dd7954ef13f03150ee9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:15:38.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.626362 22988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:38.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.627234 22988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:38.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.628072 22988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:38.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.693874 22988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.695878 22988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.65
16:15:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.767663 22988 tablet_server_runner.cc:78] Tablet server non-default flags:
16:15:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:15:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:0
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:15:38.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:38.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.769426 22988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:38.771793 22988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:38.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.816279 23003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:39.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:39.172653 22996 catalog_manager.cc:1259] Loaded cluster ID: 6e7d4b08868d47c88f9468a8da4959e1
16:15:39.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:39.173903 22996 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: loading cluster ID for follower catalog manager: success
16:15:39.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:39.217794 22996 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: acquiring CA information for follower catalog manager: success
16:15:39.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:39.253373 22996 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e60025b429fc4965b452cf68a21b35de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:15:40.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.822842 23002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:40.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:38.838912 23005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:40.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:40.619185 23004 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:15:40.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.619359 22988 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:15:40.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.621331 22988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:40.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.625708 22988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:40.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.628031 22988 hybrid_clock.cc:648] HybridClock initialized: now 1786464940627960 us; error 73 us; skew 500 ppm
16:15:40.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.630115 22988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:40.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.671183 22988 webserver.cc:460] Webserver started at http://127.22.31.65:40275/ using document root <none> and password file <none>
16:15:40.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.673159 22988 fs_manager.cc:359] Metadata directory not provided
16:15:40.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.675608 22988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:40.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.676738 22988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:15:40.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.689267 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/instance:
16:15:40.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:15:40.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:40 on dist-test-slave-9jjr"
16:15:40.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.696146 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal/instance:
16:15:40.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:15:40.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:40 on dist-test-slave-9jjr"
16:15:40.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.721161 22988 fs_manager.cc:691] Time spent creating directory manager: real 0.023s	user 0.011s	sys 0.011s
16:15:40.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.739343 23014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:40.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.741068 22988 fs_manager.cc:725] Time spent opening block manager: real 0.014s	user 0.009s	sys 0.000s
16:15:40.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.741596 22988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:40.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:15:40.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:40 on dist-test-slave-9jjr"
16:15:40.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.742244 22988 fs_report.cc:383] FS layout report
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/data
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:40.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:40.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:40.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.886500 22988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:40.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.889395 22988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:40.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.890594 22988 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:40.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.901392 22988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:15:40.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.914094 22988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:15:40.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.914462 22988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
16:15:40.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.914770 22988 ts_tablet_manager.cc:610] Registered 0 tablets
16:15:40.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:40.915000 22988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
16:15:41.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.618615 22988 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.65:36225
16:15:41.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.618719 23144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.65:36225 every 8 connection(s)
16:15:41.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.623354 22988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:15:41.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.633440 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 22988
16:15:41.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.634296 22653 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal/instance
16:15:41.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.678575 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:41.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:41.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:15:41.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:15:41.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:41.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:41.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:0
16:15:41.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:15:41.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:15:41.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:41.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:41.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:41.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:41.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:41.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:15:41.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.698945 23146 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:15:41.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.699894 23146 heartbeater.cc:464] Registering TS with master...
16:15:41.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.701344 23146 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:15:41.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.705030 22880 ts_manager.cc:194] Registered new tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:41.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.708856 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.65:38685
16:15:41.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.709226 23147 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:15:41.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.709623 23147 heartbeater.cc:464] Registering TS with master...
16:15:41.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.710747 23147 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:15:41.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.715509 22790 ts_manager.cc:194] Registered new tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:41.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.716914 23148 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:15:41.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.717406 23148 heartbeater.cc:464] Registering TS with master...
16:15:41.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.718266 23148 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:15:41.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:41.724361 22702 ts_manager.cc:194] Registered new tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:42.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.140977 23157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:42.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.141590 23157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:42.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.142081 23157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:42.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.179003 23157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:42.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.180239 23157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.66
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:42.230482 23157 tablet_server_runner.cc:78] Tablet server non-default flags:
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:0
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:42.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:42.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:42.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:42.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:42.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:42.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:15:42.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:15:42.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:42.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:42.232787 23157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:42.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:42.235442 23157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:42.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.265353 23164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:42.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:42.716616 23146 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:15:43.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.266569 23163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:43.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:42.267120 23166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:43.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:42.270265 23157 server_base.cc:1034] running on GCE node
16:15:43.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.606289 23157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:43.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.609104 23157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.610827 23157 hybrid_clock.cc:648] HybridClock initialized: now 1786464943610746 us; error 120 us; skew 500 ppm
16:15:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.611814 23157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:43.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.620725 23157 webserver.cc:460] Webserver started at http://127.22.31.66:36227/ using document root <none> and password file <none>
16:15:43.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.622184 23157 fs_manager.cc:359] Metadata directory not provided
16:15:43.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.622514 23157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:43.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.623152 23157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:15:43.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.631160 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/instance:
16:15:43.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:15:43.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:43 on dist-test-slave-9jjr"
16:15:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.632925 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal/instance:
16:15:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:15:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:43 on dist-test-slave-9jjr"
16:15:43.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.641381 23157 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.009s	sys 0.000s
16:15:43.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.648186 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:43.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.649775 23157 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
16:15:43.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.650261 23157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:15:43.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:15:43.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:43 on dist-test-slave-9jjr"
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.650624 23157 fs_report.cc:383] FS layout report
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/data
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:43.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.700132 23157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:43.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.701702 23157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:43.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.702235 23157 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:43.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.704938 23157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:15:43.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.709743 23157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:15:43.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.709975 23157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
16:15:43.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.710203 23157 ts_tablet_manager.cc:610] Registered 0 tablets
16:15:43.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.710328 23157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
16:15:43.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.908355 23157 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.66:39515
16:15:43.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.908599 23299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.66:39515 every 8 connection(s)
16:15:43.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.911180 23157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:15:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.920866 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 23157
16:15:43.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.921468 22653 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal/instance
16:15:43.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.939455 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:0
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:43.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:43.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:43.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:43.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:15:43.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.969419 23301 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:15:43.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.970311 23301 heartbeater.cc:464] Registering TS with master...
16:15:43.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.970659 23300 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:15:43.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.971033 23300 heartbeater.cc:464] Registering TS with master...
16:15:43.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.972491 23303 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:15:43.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.973176 23303 heartbeater.cc:464] Registering TS with master...
16:15:43.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.972415 23301 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:15:43.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.974498 23303 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:15:43.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.976150 23300 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:15:43.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.979547 22790 ts_manager.cc:194] Registered new tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:15:43.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.979460 22702 ts_manager.cc:194] Registered new tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:15:43.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.982014 22880 ts_manager.cc:194] Registered new tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:15:43.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:43.984664 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.66:55945
16:15:44.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.331727 23311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:44.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.332333 23311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:44.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.332973 23311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:44.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.370549 23311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:44.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.371570 23311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.67
16:15:44.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:44.410384 23311 tablet_server_runner.cc:78] Tablet server non-default flags:
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:0
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:44.411952 23311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:44.413663 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:44.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.437330 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:44.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:44.989825 23300 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:15:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.439248 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:44.438057 23321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:44.441808 23311 server_base.cc:1034] running on GCE node
16:15:45.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:45.729280 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:45.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:45.732599 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:45.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:45.734416 23311 hybrid_clock.cc:648] HybridClock initialized: now 1786464945734279 us; error 100 us; skew 500 ppm
16:15:45.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:45.735740 23311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:46.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:46.008704 23311 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-9jjr': real 0.270s	user 0.001s	sys 0.000s
16:15:46.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.013967 23311 webserver.cc:460] Webserver started at http://127.22.31.67:36945/ using document root <none> and password file <none>
16:15:46.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.014973 23311 fs_manager.cc:359] Metadata directory not provided
16:15:46.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.015173 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:46.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.015569 23311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:15:46.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.021245 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/instance:
16:15:46.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:15:46.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:46 on dist-test-slave-9jjr"
16:15:46.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.022542 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal/instance:
16:15:46.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:15:46.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:46 on dist-test-slave-9jjr"
16:15:46.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.030202 23311 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.000s
16:15:46.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.036646 23329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:46.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.037942 23311 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.001s
16:15:46.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.038342 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:46 on dist-test-slave-9jjr"
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.038937 23311 fs_report.cc:383] FS layout report
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/data
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:46.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:46.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:46.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:46.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:46.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:46.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:46.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:46.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:46.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.081791 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:46.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.083546 23311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:46.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.084100 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:46.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.086797 23311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:15:46.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.091094 23311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:15:46.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.091362 23311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
16:15:46.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.091696 23311 ts_tablet_manager.cc:610] Registered 0 tablets
16:15:46.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.091881 23311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
16:15:46.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.271947 23311 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.67:33649
16:15:46.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.272217 23455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.67:33649 every 8 connection(s)
16:15:46.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.274994 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:15:46.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.278234 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 23311
16:15:46.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.278828 22653 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal/instance
16:15:46.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.329675 23458 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:15:46.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.330260 23458 heartbeater.cc:464] Registering TS with master...
16:15:46.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.331180 23456 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:15:46.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.331563 23456 heartbeater.cc:464] Registering TS with master...
16:15:46.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.331641 23458 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:15:46.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.332589 23456 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:15:46.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.335136 22702 ts_manager.cc:194] Registered new tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:15:46.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.335397 23457 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:15:46.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.335597 22880 ts_manager.cc:194] Registered new tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:15:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.335794 23457 heartbeater.cc:464] Registering TS with master...
16:15:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.336663 23457 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:15:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.337611 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.67:59925
16:15:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.339023 22790 ts_manager.cc:194] Registered new tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:15:46.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:46.354005 22653 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
16:15:46.366 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
16:15:46.368 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

16:15:46.381 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.22.31.126"
      port: 42753
    }
    bound_http_address {
      host: "127.22.31.126"
      port: 37385
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.22.31.125"
      port: 39741
    }
    bound_http_address {
      host: "127.22.31.125"
      port: 42647
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.22.31.124"
      port: 46519
    }
    bound_http_address {
      host: "127.22.31.124"
      port: 34015
    }
  }
}

16:15:46.390 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

16:15:46.394 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.22.31.65"
      port: 36225
    }
    bound_http_address {
      host: "127.22.31.65"
      port: 40275
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.22.31.66"
      port: 39515
    }
    bound_http_address {
      host: "127.22.31.66"
      port: 36227
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.22.31.67"
      port: 33649
    }
    bound_http_address {
      host: "127.22.31.67"
      port: 36945
    }
  }
}

16:15:46.394 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
16:15:46.616 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
16:15:46.892 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.22.31.125:39741(127.22.31.125:39741)] connecting to peer
16:15:46.984 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.22.31.126:42753(127.22.31.126:42753)] connecting to peer
16:15:46.987 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.22.31.124:46519(127.22.31.124:46519)] connecting to peer
16:15:47.051 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.22.31.126:42753(127.22.31.126:42753)] Successfully connected to peer
16:15:47.053 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.22.31.125:39741(127.22.31.125:39741)] Successfully connected to peer
16:15:47.135 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.22.31.124:46519(127.22.31.124:46519)] Successfully connected to peer
16:15:47.165 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.22.31.126
16:15:47.165 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.22.31.125
16:15:47.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.343636 23456 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:15:47.390 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.22.31.124
16:15:47.419 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x9f15fdcc, L:/127.0.0.1:38740 - R:/127.22.31.126:42753] using SASL/PLAIN
16:15:47.421 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xd18994c5, L:/127.0.0.1:58088 - R:/127.22.31.125:39741] using SASL/PLAIN
16:15:47.441 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x663dca99, L:/127.0.0.1:32860 - R:/127.22.31.124:46519] using SASL/PLAIN
16:15:47.483 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
16:15:47.485 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=651), Traces: [0ms] refreshing cache from master, [38ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.31.125:39741, [143ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.31.126:42753, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.31.124:46519, [598ms] Sub RPC ConnectToMaster: received response from server master-127.22.31.126:42753: OK, [598ms] Sub RPC ConnectToMaster: received response from server master-127.22.31.125:39741: OK, [610ms] Sub RPC ConnectToMaster: received response from server master-127.22.31.124:46519: OK, deferred=null) after lookup
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.540874 22880 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32860:
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820"
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:15:47.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:15:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
16:15:47.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
16:15:47.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
16:15:47.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
16:15:47.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
16:15:47.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
16:15:47.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
16:15:47.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
16:15:47.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:47.545915 22880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
16:15:47.642 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 0149244055e14e1e9c3b3f18206abd80
16:15:47.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.668968 23226 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1c2457ffe142cea0d61a82bffe854d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820 [id=0149244055e14e1e9c3b3f18206abd80]), partition=RANGE (key) PARTITION UNBOUNDED
16:15:47.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.671307 23226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1c2457ffe142cea0d61a82bffe854d. 1 dirs total, 0 dirs full, 0 dirs failed
16:15:47.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.677502 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1c2457ffe142cea0d61a82bffe854d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820 [id=0149244055e14e1e9c3b3f18206abd80]), partition=RANGE (key) PARTITION UNBOUNDED
16:15:47.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.679544 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1c2457ffe142cea0d61a82bffe854d. 1 dirs total, 0 dirs full, 0 dirs failed
16:15:47.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.684017 23069 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1c2457ffe142cea0d61a82bffe854d (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820 [id=0149244055e14e1e9c3b3f18206abd80]), partition=RANGE (key) PARTITION UNBOUNDED
16:15:47.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.685768 23069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1c2457ffe142cea0d61a82bffe854d. 1 dirs total, 0 dirs full, 0 dirs failed
16:15:47.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.705615 23497 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap starting.
16:15:47.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.714003 23497 tablet_bootstrap.cc:654] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Neither blocks nor log segments found. Creating new log.
16:15:47.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.716045 23498 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap starting.
16:15:47.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.718204 23499 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap starting.
16:15:47.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.721428 23497 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Log is configured to *not* fsync() on all Append() calls
16:15:47.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.726933 23497 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: No bootstrap required, opened a new log
16:15:47.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.727465 23499 tablet_bootstrap.cc:654] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Neither blocks nor log segments found. Creating new log.
16:15:47.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.727535 23497 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.007s
16:15:47.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.727465 23498 tablet_bootstrap.cc:654] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Neither blocks nor log segments found. Creating new log.
16:15:47.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.730180 23498 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Log is configured to *not* fsync() on all Append() calls
16:15:47.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.730172 23499 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Log is configured to *not* fsync() on all Append() calls
16:15:47.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.736682 23498 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: No bootstrap required, opened a new log
16:15:47.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.737228 23498 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent bootstrapping tablet: real 0.022s	user 0.016s	sys 0.000s
16:15:47.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.738492 23499 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: No bootstrap required, opened a new log
16:15:47.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.739042 23499 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.004s
16:15:47.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.764891 23497 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:47.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.766012 23497 raft_consensus.cc:374] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:15:47.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.766381 23497 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f679e1cd79344040853de5b0fe6e1b38, State: Initialized, Role: FOLLOWER
16:15:47.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.767834 23497 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:47.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.773227 23497 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent starting tablet: real 0.045s	user 0.031s	sys 0.011s
16:15:47.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.779043 23498 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:47.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.780368 23498 raft_consensus.cc:374] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:15:47.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.780753 23498 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Initialized, Role: FOLLOWER
16:15:47.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.781752 23499 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:47.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.782021 23498 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:47.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.782999 23499 raft_consensus.cc:374] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:15:47.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.783411 23499 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Initialized, Role: FOLLOWER
16:15:47.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.784876 23499 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:47.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.793613 23498 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent starting tablet: real 0.056s	user 0.031s	sys 0.020s
16:15:47.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:47.797706 23462 tablet.cc:2369] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:15:47.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:47.798687 23499 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent starting tablet: real 0.059s	user 0.043s	sys 0.009s
16:15:47.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:47.931267 23304 tablet.cc:2369] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:15:47.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:47.936801 23151 tablet.cc:2369] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:15:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.075539 23504 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:15:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.076097 23504 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.078790 23504 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:48.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.096087 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38" is_pre_election: true
16:15:48.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.096035 23091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740" is_pre_election: true
16:15:48.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.096839 23091 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 0.
16:15:48.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.096920 23249 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 0.
16:15:48.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.098239 23333 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: 
16:15:48.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.099097 23504 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Leader pre-election won for term 1
16:15:48.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.099419 23504 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:15:48.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.099817 23504 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 0 FOLLOWER]: Advancing to term 1
16:15:48.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.104573 23504 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:48.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.106292 23504 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 1 election: Requested vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.107432 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:15:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.107465 23091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:15:48.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.107981 23249 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 0 FOLLOWER]: Advancing to term 1
16:15:48.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.108022 23091 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 0 FOLLOWER]: Advancing to term 1
16:15:48.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.113323 23249 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 1.
16:15:48.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.113448 23091 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 1.
16:15:48.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.114441 23333 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: 
16:15:48.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.115182 23504 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 1 FOLLOWER]: Leader election won for term 1
16:15:48.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.116938 23504 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 1 LEADER]: Becoming Leader. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Running, Role: LEADER
16:15:48.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.117933 23504 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:48.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.132699 22878 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa reported cstate change: term changed from 0 to 1, leader changed from <none> to bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67). New cstate: current_term: 1 leader_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: UNKNOWN } } }
16:15:48.245 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 0149244055e14e1e9c3b3f18206abd80
16:15:48.267 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:48.275 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299991, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:48.277 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [11ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=null) after lookup
16:15:48.289 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:48.291 [DEBUG - kudu-nio-1] (Connection.java:757) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] Successfully connected to peer
16:15:48.348 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x64d6a88a, L:/127.0.0.1:49072 - R:/127.22.31.67:33649] using TOKEN/null
16:15:48.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.364845 23091 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 1 FOLLOWER]: Refusing update from remote peer bdcfad5657c742588c55c4e90e2ac7fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:15:48.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.365039 23249 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 1 FOLLOWER]: Refusing update from remote peer bdcfad5657c742588c55c4e90e2ac7fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:15:48.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.366715 23509 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
16:15:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.367669 23504 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
16:15:48.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.396695 23520 mvcc.cc:205] Tried to move back new op lower bound from 7317360428481769472 to 7317360427499356160. Current Snapshot: MvccSnapshot[applied={T|T < 7317360428481769472}]
16:15:48.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.426065 23524 mvcc.cc:205] Tried to move back new op lower bound from 7317360428481769472 to 7317360427499356160. Current Snapshot: MvccSnapshot[applied={T|T < 7317360428481769472}]
16:15:48.579 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:15:48.581 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.67:33649
16:15:48.583 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:15:48.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:48.584033 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 23311
16:15:48.626 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:15:48.631 [DEBUG - kudu-nio-1] (Connection.java:239) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] handling channelInactive
16:15:48.632 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state READY due to: connection closed
16:15:48.633 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: connection closed
16:15:48.664 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:48.665 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:48.666 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test)) after lookup
16:15:48.667 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:48.668 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.668 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.669 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.680 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:48.680 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:48.681 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=54), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [42ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [43ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [53ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237))) after lookup
16:15:48.681 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:48.683 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.683 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.684 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.741 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:48.742 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:48.743 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=115), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [42ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [43ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [53ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [55ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [58ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [58ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [114ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136))) after lookup
16:15:48.744 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:48.746 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.746 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.746 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.841 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:48.841 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:48.842 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=215), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [42ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [43ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [53ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [55ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [58ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [58ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [114ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [118ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [120ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [120ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [208ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [214ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762))) after lookup
16:15:48.843 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:48.845 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.846 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:48.846 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.260 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:49.261 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:49.262 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=634), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [42ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [43ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [53ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [55ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [58ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [58ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [114ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [118ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [120ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [120ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [208ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [214ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [217ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [220ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [220ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [629ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [633ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545))) after lookup
16:15:49.263 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:49.264 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.264 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.264 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.759 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:49.760 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:49.761 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1133), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [42ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [43ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [53ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [55ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [58ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [58ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [114ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [118ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [120ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [120ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [208ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [214ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [217ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [220ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [220ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [629ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [633ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [636ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [638ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [638ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1128ms] refreshing cache from master, [1128ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1132ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545) -> (continuation of Deferred@1863524298 after retry RPC@147088498), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545) -> (continuation of Deferred@1863524298 after retry RPC@147088498))) after lookup
16:15:49.762 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:49.764 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.764 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.764 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.800 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:49.801 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:49.802 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1174), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [42ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [43ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [53ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [55ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [58ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [58ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [114ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [118ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [120ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [120ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [208ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [214ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [217ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [220ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [220ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [629ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [633ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [636ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [638ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [638ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1128ms] refreshing cache from master, [1128ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1132ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [1135ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [1138ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1138ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1168ms] refreshing cache from master, [1168ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1173ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545) -> (continuation of Deferred@1863524298 after retry RPC@147088498) -> (continuation of Deferred@19087834 after retry RPC@711934584), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545) -> (continuation of Deferred@1863524298 after retry RPC@147088498) -> (continuation of Deferred@19087834 after retry RPC@711934584))) after lookup
16:15:49.803 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:15:49.805 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.805 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:49.805 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:15:50.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.015515 23536 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:15:50.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.016000 23536 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:50.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.017081 23537 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:15:50.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.018224 23537 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:50.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.019678 23536 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:50.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.028529 23537 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:15:50.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:50.038872 23176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:15:50.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:50.047644 23016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:15:50.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:50.049115 23176 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:15:50.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:50.053845 23016 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:15:50.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.057247 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38" is_pre_election: true
16:15:50.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.058043 23249 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 1.
16:15:50.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.059459 23015 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:15:50.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.060732 23537 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 1 FOLLOWER]: Leader pre-election won for term 2
16:15:50.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.061125 23537 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:15:50.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.061599 23537 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 1 FOLLOWER]: Advancing to term 2
16:15:50.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.061113 23091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "f679e1cd79344040853de5b0fe6e1b38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740" is_pre_election: true
16:15:50.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.067421 23537 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:50.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.068532 23091 raft_consensus.cc:2379] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f679e1cd79344040853de5b0fe6e1b38 in current term 2: Already voted for candidate 3de21ca168134ac9a117e9e6cfc3a740 in this term.
16:15:50.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.069487 23537 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 2 election: Requested vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:15:50.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.071309 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:15:50.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.071982 23249 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 1 FOLLOWER]: Advancing to term 2
16:15:50.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:50.074681 23016 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:15:50.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.075941 23178 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f679e1cd79344040853de5b0fe6e1b38; no voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa
16:15:50.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.078889 23249 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 2.
16:15:50.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.079437 23536 raft_consensus.cc:2726] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
16:15:50.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.080094 23015 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:15:50.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.080794 23537 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 FOLLOWER]: Leader election won for term 2
16:15:50.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.082867 23537 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 LEADER]: Becoming Leader. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Running, Role: LEADER
16:15:50.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.084858 23537 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:50.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.096602 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 reported cstate change: term changed from 1 to 2, leader changed from bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67) to 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65). New cstate: current_term: 2 leader_uuid: "3de21ca168134ac9a117e9e6cfc3a740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: HEALTHY } } }
16:15:50.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:50.499789 23016 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 -> Peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Couldn't send request to peer bdcfad5657c742588c55c4e90e2ac7fa. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:15:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.678452 23249 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Refusing update from remote peer 3de21ca168134ac9a117e9e6cfc3a740: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
16:15:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:50.679919 23537 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
16:15:50.901 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:50.901 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:50.903 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=16, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2275), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [7ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: connection closed, [7ms] delaying RPC due to: Network error: connection closed, [28ms] refreshing cache from master, [29ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [42ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [43ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [49ms] refreshing cache from master, [49ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [53ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [55ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [58ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [58ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [109ms] refreshing cache from master, [109ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [114ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [118ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [120ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [120ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [208ms] refreshing cache from master, [209ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [214ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [217ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [220ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [220ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [629ms] refreshing cache from master, [629ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [633ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [636ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [638ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [638ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1128ms] refreshing cache from master, [1128ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1132ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [1135ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [1138ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1138ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1168ms] refreshing cache from master, [1168ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1173ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [1177ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [1179ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1179ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2268ms] refreshing cache from master, [2269ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [2273ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@2045172486(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@2b678488 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545) -> (continuation of Deferred@1863524298 after retry RPC@147088498) -> (continuation of Deferred@19087834 after retry RPC@711934584) -> (continuation of Deferred@332428044 after retry RPC@798948185), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78e6f12 -> wakeup thread Time-limited test -> (continuation of Deferred@802652931 after retry RPC@289712237) -> (continuation of Deferred@1721093983 after retry RPC@1378223136) -> (continuation of Deferred@1175092072 after retry RPC@1981137762) -> (continuation of Deferred@697759418 after retry RPC@1900031545) -> (continuation of Deferred@1863524298 after retry RPC@147088498) -> (continuation of Deferred@19087834 after retry RPC@711934584) -> (continuation of Deferred@332428044 after retry RPC@798948185))) after lookup
16:15:50.904 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:50.906 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] Successfully connected to peer
16:15:50.939 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xaa528009, L:/127.0.0.1:48062 - R:/127.22.31.65:36225] using TOKEN/null
16:15:51.031 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.67:33649
16:15:51.034 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:51.036619 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:15:51.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:15:51.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:51.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:15:51.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:15:51.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:51.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:51.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:51.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:15:51.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.407747 23573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:51.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.408222 23573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:51.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.408664 23573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:51.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.444871 23573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:51.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.445793 23573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.67
16:15:51.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:51.482187 23573 tablet_server_runner.cc:78] Tablet server non-default flags:
16:15:51.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:51.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:51.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:51.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:51.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:51.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:51.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:51.483841 23573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:51.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:51.485611 23573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:51.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.509894 23583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:52.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.512075 23582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:52.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:51.510283 23585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:52.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:52.773375 23584 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:15:52.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.773401 23573 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:15:52.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.778851 23573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:52.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.781335 23573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.782827 23573 hybrid_clock.cc:648] HybridClock initialized: now 1786464952782782 us; error 43 us; skew 500 ppm
16:15:52.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.783944 23573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:52.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.795181 23573 webserver.cc:460] Webserver started at http://127.22.31.67:36945/ using document root <none> and password file <none>
16:15:52.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.796334 23573 fs_manager.cc:359] Metadata directory not provided
16:15:52.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.796624 23573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:52.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.805337 23573 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.006s
16:15:52.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.810797 23597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.812002 23573 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
16:15:52.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.812444 23573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:52.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:15:52.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:46 on dist-test-slave-9jjr"
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.815430 23573 fs_report.cc:383] FS layout report
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/data
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:52.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:52.850485 23016 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 -> Peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Couldn't send request to peer bdcfad5657c742588c55c4e90e2ac7fa. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:15:52.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.876639 23573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:52.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.878360 23573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:52.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.878954 23573 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.881903 23573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:15:52.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.888856 23604 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:15:52.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.897361 23573 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:15:52.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.897722 23573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.001s	sys 0.001s
16:15:52.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.898048 23573 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:15:52.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.903364 23573 ts_tablet_manager.cc:610] Registered 1 tablets
16:15:52.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.903620 23573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
16:15:52.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.904238 23604 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap starting.
16:15:52.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:52.973443 23604 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Log is configured to *not* fsync() on all Append() calls
16:15:53.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.031692 23604 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:15:53.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.033427 23604 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap complete.
16:15:53.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.036116 23604 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent bootstrapping tablet: real 0.132s	user 0.078s	sys 0.042s
16:15:53.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.066170 23604 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.068945 23604 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Initialized, Role: FOLLOWER
16:15:53.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.070230 23604 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:53.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.075480 23604 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent starting tablet: real 0.039s	user 0.024s	sys 0.012s
16:15:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.152810 23573 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.67:33649
16:15:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.153111 23724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.67:33649 every 8 connection(s)
16:15:53.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.155975 23573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:15:53.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.161926 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 23573
16:15:53.163 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:15:53.207 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:15:53.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.208949 23725 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:15:53.214 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.65:36225
16:15:53.215 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:15:53.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.209568 23725 heartbeater.cc:464] Registering TS with master...
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.211289 23725 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.215498 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 22988
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.217042 23728 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.217080 22880 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.217445 23728 heartbeater.cc:464] Registering TS with master...
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.218417 23728 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.218683 23727 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.218973 23727 heartbeater.cc:464] Registering TS with master...
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.219959 23727 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:15:53.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.222033 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.67:47301
16:15:53.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.223835 22789 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:15:53.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:53.223759 22702 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:15:53.259 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] handling channelInactive
16:15:53.259 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:15:53.260 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state READY due to: connection closed
16:15:53.260 [DEBUG - Time-limited test] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:53.262 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.262 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.262 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.280 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:53.281 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:53.281 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [2ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1232217162(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@6a6beaa9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b38d85c -> wakeup thread Time-limited test)) after lookup
16:15:53.282 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:53.284 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.284 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.285 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.300 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:53.300 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:53.301 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [2ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [22ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [24ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [25ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1232217162(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@6a6beaa9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b38d85c -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729))) after lookup
16:15:53.301 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:53.302 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.303 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.303 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.320 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:53.321 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:53.323 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [2ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [22ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [24ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [25ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [43ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [59ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1232217162(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@6a6beaa9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b38d85c -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340))) after lookup
16:15:53.324 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:53.327 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.327 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.328 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.460 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:53.460 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:53.461 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=200), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [2ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [22ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [24ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [25ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [43ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [59ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [63ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [67ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [68ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [194ms] refreshing cache from master, [194ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [199ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1232217162(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@6a6beaa9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b38d85c -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402))) after lookup
16:15:53.462 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:53.463 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.463 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.463 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.520 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:53.521 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:53.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=261), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [2ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [22ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [24ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [25ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [43ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [59ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [63ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [67ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [68ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [194ms] refreshing cache from master, [194ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [199ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [201ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [203ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [203ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [255ms] refreshing cache from master, [255ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [259ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1232217162(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@6a6beaa9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402) -> (continuation of Deferred@785026958 after retry RPC@1328439671), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b38d85c -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402) -> (continuation of Deferred@785026958 after retry RPC@1328439671))) after lookup
16:15:53.523 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:53.525 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.525 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:53.525 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:54.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.228507 23725 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:15:54.260 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:54.260 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:15:54.261 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1000), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [2ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [22ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [24ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [25ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [43ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [59ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [63ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [67ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [68ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [194ms] refreshing cache from master, [194ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [199ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [201ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [203ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [203ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [255ms] refreshing cache from master, [255ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [259ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [262ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [265ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [265ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [994ms] refreshing cache from master, [994ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [998ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1232217162(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@6a6beaa9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402) -> (continuation of Deferred@785026958 after retry RPC@1328439671) -> (continuation of Deferred@1178906303 after retry RPC@2061327065), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b38d85c -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402) -> (continuation of Deferred@785026958 after retry RPC@1328439671) -> (continuation of Deferred@1178906303 after retry RPC@2061327065))) after lookup
16:15:54.262 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:15:54.263 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:54.264 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:54.264 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:15:54.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.356426 23738 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3de21ca168134ac9a117e9e6cfc3a740)
16:15:54.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.357174 23738 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:54.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.359366 23738 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:54.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:54.368829 23178 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:15:54.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.375389 23674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "f679e1cd79344040853de5b0fe6e1b38" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" is_pre_election: true
16:15:54.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.376165 23674 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f679e1cd79344040853de5b0fe6e1b38 in term 1.
16:15:54.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.377655 23176 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38; no voters: 3de21ca168134ac9a117e9e6cfc3a740
16:15:54.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.378508 23738 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Leader pre-election won for term 3
16:15:54.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.378857 23738 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3de21ca168134ac9a117e9e6cfc3a740)
16:15:54.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.379094 23738 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 2 FOLLOWER]: Advancing to term 3
16:15:54.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.384533 23738 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:54.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.386243 23738 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 3 election: Requested vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:54.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.387281 23674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "f679e1cd79344040853de5b0fe6e1b38" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:15:54.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.387908 23674 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 1 FOLLOWER]: Advancing to term 3
16:15:54.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:54.390341 23178 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:15:54.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.395597 23674 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f679e1cd79344040853de5b0fe6e1b38 in term 3.
16:15:54.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.396772 23176 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38; no voters: 3de21ca168134ac9a117e9e6cfc3a740
16:15:54.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.397598 23738 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 3 FOLLOWER]: Leader election won for term 3
16:15:54.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.398938 23738 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 3 LEADER]: Becoming Leader. State: Replica: f679e1cd79344040853de5b0fe6e1b38, State: Running, Role: LEADER
16:15:54.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.400033 23738 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:54.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:54.409548 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 reported cstate change: term changed from 2 to 3, leader changed from 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65) to f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66). New cstate: current_term: 3 leader_uuid: "f679e1cd79344040853de5b0fe6e1b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: UNKNOWN } } }
16:15:54.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:54.814819 23178 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 -> Peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Couldn't send request to peer 3de21ca168134ac9a117e9e6cfc3a740. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:15:55.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:55.009838 23674 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 3 FOLLOWER]: Refusing update from remote peer f679e1cd79344040853de5b0fe6e1b38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
16:15:55.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:55.011466 23738 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
16:15:55.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:55.237855 23178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111) [suppressed 3 similar messages]
16:15:55.541 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:55.541 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:15:55.542 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2282), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [2ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [22ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [24ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [25ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [43ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [59ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [63ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [67ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [68ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [194ms] refreshing cache from master, [194ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [199ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [201ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [203ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [203ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [255ms] refreshing cache from master, [255ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [259ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [262ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [265ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [265ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [994ms] refreshing cache from master, [994ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [998ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [1002ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [1004ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [1004ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [2274ms] refreshing cache from master, [2275ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [2280ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1232217162(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@6a6beaa9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402) -> (continuation of Deferred@785026958 after retry RPC@1328439671) -> (continuation of Deferred@1178906303 after retry RPC@2061327065) -> (continuation of Deferred@1514878520 after retry RPC@455277418), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b38d85c -> wakeup thread Time-limited test -> (continuation of Deferred@1014814306 after retry RPC@1633358729) -> (continuation of Deferred@1952649097 after retry RPC@1521756340) -> (continuation of Deferred@3806719 after retry RPC@187160402) -> (continuation of Deferred@785026958 after retry RPC@1328439671) -> (continuation of Deferred@1178906303 after retry RPC@2061327065) -> (continuation of Deferred@1514878520 after retry RPC@455277418))) after lookup
16:15:55.543 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:55.545 [DEBUG - kudu-nio-1] (Connection.java:757) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] Successfully connected to peer
16:15:55.569 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x239da70e, L:/127.0.0.1:57544 - R:/127.22.31.66:39515] using TOKEN/null
16:15:55.639 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.65:36225
16:15:55.641 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:55.642827 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:55.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:36225
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40275
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:55.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:15:56.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.020370 23774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:15:56.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.020889 23774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:15:56.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.021368 23774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:15:56.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.061408 23774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:15:56.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.062912 23774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.65
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:56.104799 23774 tablet_server_runner.cc:78] Tablet server non-default flags:
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:36225
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:15:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40275
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:15:56.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:56.106173 23774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:15:56.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:56.108124 23774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:15:56.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.132048 23784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:57.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:57.354646 23178 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 -> Peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Couldn't send request to peer 3de21ca168134ac9a117e9e6cfc3a740. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:15:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.132988 23788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:57.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:56.132992 23785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:15:57.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:57.396713 23786 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:15:57.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.396842 23774 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:15:57.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.400617 23774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:15:57.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.403775 23774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:15:57.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.405267 23774 hybrid_clock.cc:648] HybridClock initialized: now 1786464957405225 us; error 49 us; skew 500 ppm
16:15:57.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.406116 23774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:15:57.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.413453 23774 webserver.cc:460] Webserver started at http://127.22.31.65:40275/ using document root <none> and password file <none>
16:15:57.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.414462 23774 fs_manager.cc:359] Metadata directory not provided
16:15:57.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.414674 23774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:15:57.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.422796 23774 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.004s
16:15:57.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.428177 23801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:15:57.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.429768 23774 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.003s
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.430099 23774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:40 on dist-test-slave-9jjr"
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.432926 23774 fs_report.cc:383] FS layout report
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/data
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:15:57.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:15:57.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.475683 23774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:15:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.477252 23774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:15:57.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.477679 23774 kserver.cc:163] Server-wide thread pool size limit: 3276
16:15:57.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.480582 23774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:15:57.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.486965 23809 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:15:57.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.495399 23774 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:15:57.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.495728 23774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
16:15:57.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.495977 23774 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:15:57.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.501426 23774 ts_tablet_manager.cc:610] Registered 1 tablets
16:15:57.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.501627 23774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.002s	sys 0.001s
16:15:57.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.502548 23809 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap starting.
16:15:57.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.564150 23809 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Log is configured to *not* fsync() on all Append() calls
16:15:57.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.609941 23809 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:15:57.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.611169 23809 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap complete.
16:15:57.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.612939 23809 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent bootstrapping tablet: real 0.111s	user 0.071s	sys 0.036s
16:15:57.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.642516 23809 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:57.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.645812 23809 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Initialized, Role: FOLLOWER
16:15:57.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.647063 23809 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:57.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.650943 23809 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent starting tablet: real 0.038s	user 0.029s	sys 0.010s
16:15:57.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.696246 23774 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.65:36225
16:15:57.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.696465 23931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.65:36225 every 8 connection(s)
16:15:57.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.698987 23774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:15:57.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.705375 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 23774
16:15:57.706 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:15:57.747 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:15:57.749 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.66:39515
16:15:57.749 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:15:57.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.750480 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 23157
16:15:57.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.785184 23934 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:15:57.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.785627 23934 heartbeater.cc:464] Registering TS with master...
16:15:57.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.786924 23934 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:15:57.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.792860 22790 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:57.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.796309 23932 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:15:57.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.796774 23932 heartbeater.cc:464] Registering TS with master...
16:15:57.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.797849 23932 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:15:57.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.801673 22878 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:57.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.806262 23935 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:15:57.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.806378 22878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.65:40081
16:15:57.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.806680 23935 heartbeater.cc:464] Registering TS with master...
16:15:57.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.807631 23935 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:15:57.809 [DEBUG - kudu-nio-1] (Connection.java:239) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] handling channelInactive
16:15:57.809 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state READY due to: connection closed
16:15:57.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:57.811914 22702 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:57.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:57.812350 23944 negotiation.cc:337] Failed RPC negotiation. Trace:
16:15:57.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.763902 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.31.66:41599 (local address 127.22.31.65:36225)
16:15:57.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.772374 (+  8472us) server_negotiation.cc:206] Beginning negotiation
16:15:57.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.772401 (+    27us) server_negotiation.cc:399] Waiting for connection header
16:15:57.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.772478 (+    77us) server_negotiation.cc:407] Connection header received
16:15:57.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.772738 (+   260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:15:57.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.772771 (+    33us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:15:57.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.773248 (+   477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:15:57.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.773561 (+   313us) server_negotiation.cc:226] Negotiated authn=SASL
16:15:57.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:15:57.809585 (+ 36024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.31.66:41599: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:15:57.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":7837,"thread_start_us":10101,"threads_started":1}
16:15:57.815 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:15:57.816 [DEBUG - Time-limited test] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:57.818 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.818 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.819 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.840 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:57.841 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:15:57.841 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [4ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [24ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1034334049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@701c1118 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76a558c6 -> wakeup thread Time-limited test)) after lookup
16:15:57.842 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:57.844 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.844 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.845 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.860 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:57.861 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:15:57.861 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [4ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [24ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [30ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [30ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [45ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1034334049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@701c1118 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76a558c6 -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745))) after lookup
16:15:57.862 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:57.863 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.864 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.864 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.900 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:57.900 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:15:57.901 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [4ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [24ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [30ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [30ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [45ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [47ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [49ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [49ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [84ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1034334049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@701c1118 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76a558c6 -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467))) after lookup
16:15:57.902 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:57.904 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.904 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:57.904 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.000 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:58.001 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:15:58.001 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=185), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [4ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [24ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [30ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [30ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [45ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [47ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [49ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [49ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [84ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [86ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [89ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [89ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [180ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [185ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1034334049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@701c1118 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76a558c6 -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758))) after lookup
16:15:58.002 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:58.003 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.005 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.005 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.059 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:58.059 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:15:58.060 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=244), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [4ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [24ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [30ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [30ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [45ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [47ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [49ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [49ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [84ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [86ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [89ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [89ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [180ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [185ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [187ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [190ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [190ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [239ms] refreshing cache from master, [240ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [244ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1034334049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@701c1118 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758) -> (continuation of Deferred@1854511293 after retry RPC@755535834), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76a558c6 -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758) -> (continuation of Deferred@1854511293 after retry RPC@755535834))) after lookup
16:15:58.060 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:58.062 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.062 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.063 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.399 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:15:58.399 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:15:58.399 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=583), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [4ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [24ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [30ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [30ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [45ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [47ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [49ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [49ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [84ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [86ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [89ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [89ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [180ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [185ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [187ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [190ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [190ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [239ms] refreshing cache from master, [240ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [244ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [245ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [248ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [248ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [579ms] refreshing cache from master, [579ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [583ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1034334049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@701c1118 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758) -> (continuation of Deferred@1854511293 after retry RPC@755535834) -> (continuation of Deferred@1536950574 after retry RPC@847459288), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76a558c6 -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758) -> (continuation of Deferred@1854511293 after retry RPC@755535834) -> (continuation of Deferred@1536950574 after retry RPC@847459288))) after lookup
16:15:58.400 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:15:58.401 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.401 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.401 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:15:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.707867 23945 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f679e1cd79344040853de5b0fe6e1b38)
16:15:58.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.708379 23945 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:58.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.710918 23945 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:58.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:58.725361 23598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:15:58.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:58.731151 23598 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:15:58.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.739421 23880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740" is_pre_election: true
16:15:58.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.740757 23880 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 2.
16:15:58.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.742681 23601 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: f679e1cd79344040853de5b0fe6e1b38
16:15:58.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.743618 23945 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 3 FOLLOWER]: Leader pre-election won for term 4
16:15:58.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.744006 23945 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 3 FOLLOWER]: Starting leader election (detected failure of leader f679e1cd79344040853de5b0fe6e1b38)
16:15:58.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.744421 23945 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 3 FOLLOWER]: Advancing to term 4
16:15:58.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.750422 23945 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:58.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.752127 23945 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 4 election: Requested vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:15:58.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.753486 23880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:15:58.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.754251 23880 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 2 FOLLOWER]: Advancing to term 4
16:15:58.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:58.756464 23598 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:15:58.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.762924 23880 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 4.
16:15:58.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.764026 23601 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: f679e1cd79344040853de5b0fe6e1b38
16:15:58.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.764745 23945 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 4 FOLLOWER]: Leader election won for term 4
16:15:58.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.766531 23945 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 4 LEADER]: Becoming Leader. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Running, Role: LEADER
16:15:58.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.767302 23945 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:15:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.777839 22878 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa reported cstate change: term changed from 3 to 4, leader changed from f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66) to bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67). New cstate: current_term: 4 leader_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: UNKNOWN } } }
16:15:58.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:58.815941 23932 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:15:59.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:59.237231 23880 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 4 FOLLOWER]: Refusing update from remote peer bdcfad5657c742588c55c4e90e2ac7fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
16:15:59.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:15:59.239419 23945 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
16:15:59.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:15:59.292917 23598 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa -> Peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Couldn't send request to peer f679e1cd79344040853de5b0fe6e1b38. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:00.721 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:00.722 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:00.722 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2906), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [4ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [24ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [30ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [30ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [39ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [45ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [47ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [49ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [49ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [84ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [86ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [89ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [89ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [180ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [185ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [187ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [190ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [190ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [239ms] refreshing cache from master, [240ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [244ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [245ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [248ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [248ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [579ms] refreshing cache from master, [579ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [583ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [585ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [586ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [586ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [2900ms] refreshing cache from master, [2900ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [2906ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1034334049(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@701c1118 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758) -> (continuation of Deferred@1854511293 after retry RPC@755535834) -> (continuation of Deferred@1536950574 after retry RPC@847459288) -> (continuation of Deferred@1543422907 after retry RPC@1125602828), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76a558c6 -> wakeup thread Time-limited test -> (continuation of Deferred@1314676836 after retry RPC@316895745) -> (continuation of Deferred@895191030 after retry RPC@722640467) -> (continuation of Deferred@1220303579 after retry RPC@246903758) -> (continuation of Deferred@1854511293 after retry RPC@755535834) -> (continuation of Deferred@1536950574 after retry RPC@847459288) -> (continuation of Deferred@1543422907 after retry RPC@1125602828))) after lookup
16:16:00.723 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:00.725 [DEBUG - kudu-nio-1] (Connection.java:757) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] Successfully connected to peer
16:16:00.755 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xe6da92d5, L:/127.0.0.1:34502 - R:/127.22.31.67:33649] using TOKEN/null
16:16:00.822 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.66:39515
16:16:00.825 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:16:00.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:00.827417 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:00.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:39515
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36227
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:00.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.207877 23990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.208657 23990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:01.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.209358 23990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:01.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.244887 23990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:01.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.246007 23990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.66
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:01.287710 23990 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:39515
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36227
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:01.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:01.288998 23990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:01.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:01.290793 23990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:01.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.315833 24001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:01.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.568200 23598 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa -> Peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Couldn't send request to peer f679e1cd79344040853de5b0fe6e1b38. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:16:02.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.315876 23999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:02.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:01.317567 23990 server_base.cc:1034] running on GCE node
16:16:02.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:01.315884 23998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:02.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.590494 23990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:02.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.592865 23990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:02.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.594415 23990 hybrid_clock.cc:648] HybridClock initialized: now 1786464962594391 us; error 65 us; skew 500 ppm
16:16:02.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.595263 23990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:02.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.601965 23990 webserver.cc:460] Webserver started at http://127.22.31.66:36227/ using document root <none> and password file <none>
16:16:02.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.603010 23990 fs_manager.cc:359] Metadata directory not provided
16:16:02.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.603266 23990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:02.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.611768 23990 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.000s	sys 0.004s
16:16:02.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.616649 24010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:02.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.618002 23990 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
16:16:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.618602 23990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:16:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:43 on dist-test-slave-9jjr"
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.621304 23990 fs_report.cc:383] FS layout report
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/data
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:02.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.680917 23990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:02.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.682713 23990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:02.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.683279 23990 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:02.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.686096 23990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:02.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.692444 24018 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:02.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.701333 23990 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:02.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.701591 23990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
16:16:02.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.701819 23990 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:02.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.710012 23990 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:02.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.710358 23990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s	user 0.009s	sys 0.000s
16:16:02.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.711179 24018 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap starting.
16:16:02.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.788257 24018 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Log is configured to *not* fsync() on all Append() calls
16:16:02.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.860093 24018 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:02.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.861123 24018 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap complete.
16:16:02.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.863385 24018 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent bootstrapping tablet: real 0.153s	user 0.098s	sys 0.044s
16:16:02.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.893350 24018 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:02.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.896973 24018 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f679e1cd79344040853de5b0fe6e1b38, State: Initialized, Role: FOLLOWER
16:16:02.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.898319 24018 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:02.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.903673 24018 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent starting tablet: real 0.040s	user 0.039s	sys 0.000s
16:16:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.916947 23990 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.66:39515
16:16:02.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.917101 24137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.66:39515 every 8 connection(s)
16:16:02.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.919698 23990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:02.927 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:02.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.924724 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 23990
16:16:02.957 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:02.963 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.67:33649
16:16:02.964 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:16:02.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.964814 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 23573
16:16:02.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.989405 24138 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.989964 24138 heartbeater.cc:464] Registering TS with master...
16:16:02.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.991354 24138 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:02.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.996024 22880 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:03.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:02.999918 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.66:40051
16:16:03.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.008677 24139 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:03.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.009148 24139 heartbeater.cc:464] Registering TS with master...
16:16:03.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.010030 24139 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:03.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.013988 22790 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:03.019 [DEBUG - kudu-nio-1] (Connection.java:239) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] handling channelInactive
16:16:03.020 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state READY due to: connection closed
16:16:03.035 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:03.036 [DEBUG - Time-limited test] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:03.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.037107 24140 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:03.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.037530 24140 heartbeater.cc:464] Registering TS with master...
16:16:03.038 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.038 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.038295 24140 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:03.039 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:03.042433 22702 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:03.059 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:03.060 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:03.060 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [3ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [3ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [23ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@984074069(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@cbab396 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60a94e08 -> wakeup thread Time-limited test)) after lookup
16:16:03.061 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:03.063 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.064 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.064 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.079 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:03.079 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:03.080 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [3ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [3ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [23ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [25ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@984074069(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@cbab396 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60a94e08 -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753))) after lookup
16:16:03.080 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:03.082 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.082 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.082 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.119 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:03.119 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:03.120 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [3ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [3ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [23ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [25ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [44ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [46ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [46ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [82ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@984074069(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@cbab396 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60a94e08 -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607))) after lookup
16:16:03.121 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:03.123 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.123 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.123 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.159 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:03.159 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:03.160 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=124), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [3ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [3ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [23ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [25ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [44ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [46ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [46ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [82ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [85ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [87ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [87ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [118ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [123ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@984074069(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@cbab396 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607) -> (continuation of Deferred@1632622367 after retry RPC@876068242), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60a94e08 -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607) -> (continuation of Deferred@1632622367 after retry RPC@876068242))) after lookup
16:16:03.161 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:03.162 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.162 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.162 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.643 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:03.644 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:03.644 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=608), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [3ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [3ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [23ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [25ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [44ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [46ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [46ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [82ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [85ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [87ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [87ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [118ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [123ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [124ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [126ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [126ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [578ms] refreshing cache from master, [579ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [607ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@984074069(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@cbab396 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607) -> (continuation of Deferred@1632622367 after retry RPC@876068242) -> (continuation of Deferred@2664828 after retry RPC@1578979608), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60a94e08 -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607) -> (continuation of Deferred@1632622367 after retry RPC@876068242) -> (continuation of Deferred@2664828 after retry RPC@1578979608))) after lookup
16:16:03.646 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:03.647 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.648 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:03.648 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:04.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.005429 24138 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:04.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.294476 24151 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:16:04.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.294883 24151 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:04.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.297248 24151 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:04.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:04.301213 23803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:16:04.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:04.304652 23803 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:16:04.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.313019 24087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38" is_pre_election: true
16:16:04.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.313817 24087 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 3.
16:16:04.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.315359 23802 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:16:04.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.316217 24151 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 4 FOLLOWER]: Leader pre-election won for term 5
16:16:04.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.316463 24151 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:16:04.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.316699 24151 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 4 FOLLOWER]: Advancing to term 5
16:16:04.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.321400 24151 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:04.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.323434 24151 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 5 election: Requested vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:04.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.324580 24087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:16:04.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.325209 24087 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 3 FOLLOWER]: Advancing to term 5
16:16:04.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:04.327296 23803 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:16:04.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.332865 24087 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 5.
16:16:04.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.333940 23802 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:16:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.334847 24151 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 FOLLOWER]: Leader election won for term 5
16:16:04.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.336061 24151 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 LEADER]: Becoming Leader. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Running, Role: LEADER
16:16:04.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.336879 24151 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:04.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.346253 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 reported cstate change: term changed from 4 to 5, leader changed from bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67) to 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65). New cstate: current_term: 5 leader_uuid: "3de21ca168134ac9a117e9e6cfc3a740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: HEALTHY } } }
16:16:04.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:04.731032 23803 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 -> Peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Couldn't send request to peer bdcfad5657c742588c55c4e90e2ac7fa. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.877269 24087 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 5 FOLLOWER]: Refusing update from remote peer 3de21ca168134ac9a117e9e6cfc3a740: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
16:16:04.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:04.879179 24151 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
16:16:04.900 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:04.901 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:04.902 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1865), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [3ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [3ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [18ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [23ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [25ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [44ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [46ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [46ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [82ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [85ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [87ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [87ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [118ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [123ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [124ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [126ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [126ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [578ms] refreshing cache from master, [579ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [607ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [609ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [612ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [612ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1859ms] refreshing cache from master, [1859ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1863ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@984074069(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@cbab396 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607) -> (continuation of Deferred@1632622367 after retry RPC@876068242) -> (continuation of Deferred@2664828 after retry RPC@1578979608) -> (continuation of Deferred@84369224 after retry RPC@832985953), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@60a94e08 -> wakeup thread Time-limited test -> (continuation of Deferred@1551131617 after retry RPC@1988603753) -> (continuation of Deferred@1364227244 after retry RPC@1077060607) -> (continuation of Deferred@1632622367 after retry RPC@876068242) -> (continuation of Deferred@2664828 after retry RPC@1578979608) -> (continuation of Deferred@84369224 after retry RPC@832985953))) after lookup
16:16:04.903 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:04.905 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] Successfully connected to peer
16:16:04.963 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x77916b47, L:/127.0.0.1:39994 - R:/127.22.31.65:36225] using TOKEN/null
16:16:05.046 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.67:33649
16:16:05.046 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:16:05.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:05.048417 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:05.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:05.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:05.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:16:05.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:16:05.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:16:05.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:05.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:05.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:05.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:05.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.404765 24183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:05.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.405268 24183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:05.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.405733 24183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:05.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.442512 24183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:05.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.443728 24183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.67
16:16:05.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:05.483199 24183 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:05.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:05.484596 24183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:05.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:05.486337 24183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:05.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.507110 24198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:06.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.507145 24195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:06.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:05.508659 24183 server_base.cc:1034] running on GCE node
16:16:06.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:05.507627 24194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:06.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.811136 24183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:06.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.815277 24183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:06.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.816838 24183 hybrid_clock.cc:648] HybridClock initialized: now 1786464966816756 us; error 56 us; skew 500 ppm
16:16:06.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.818295 24183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:06.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.828169 24183 webserver.cc:460] Webserver started at http://127.22.31.67:36945/ using document root <none> and password file <none>
16:16:06.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.830076 24183 fs_manager.cc:359] Metadata directory not provided
16:16:06.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.830520 24183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:06.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.843627 24183 fs_manager.cc:709] Time spent opening directory manager: real 0.008s	user 0.003s	sys 0.005s
16:16:06.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.850440 24212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:06.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.853258 24183 fs_manager.cc:725] Time spent opening block manager: real 0.006s	user 0.003s	sys 0.001s
16:16:06.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.853772 24183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:06.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:16:06.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:46 on dist-test-slave-9jjr"
16:16:06.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.858597 24183 fs_report.cc:383] FS layout report
16:16:06.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/data
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:06.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.908018 24183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:06.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.909619 24183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:06.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.910058 24183 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:06.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.912963 24183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:06.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.920928 24219 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:06.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.934681 24183 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:06.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.935019 24183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s	user 0.002s	sys 0.000s
16:16:06.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.935336 24183 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:06.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.943281 24183 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:06.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.943588 24183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s	user 0.008s	sys 0.000s
16:16:06.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:06.944315 24219 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap starting.
16:16:07.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.020936 24219 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Log is configured to *not* fsync() on all Append() calls
16:16:07.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.123404 24219 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:07.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.124439 24219 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap complete.
16:16:07.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.126418 24219 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent bootstrapping tablet: real 0.183s	user 0.115s	sys 0.057s
16:16:07.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.151419 24219 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:07.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.155273 24219 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Initialized, Role: FOLLOWER
16:16:07.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.156922 24219 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:07.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.162921 24219 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent starting tablet: real 0.036s	user 0.027s	sys 0.009s
16:16:07.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.215029 24183 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.67:33649
16:16:07.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.215178 24339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.67:33649 every 8 connection(s)
16:16:07.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.217584 24183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:07.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.219058 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 24183
16:16:07.220 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:07.237 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
16:16:07.242 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
16:16:07.267 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:07.267 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.65:36225
16:16:07.268 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:16:07.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.268646 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 23774
16:16:07.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.279632 24340 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:07.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.280223 24340 heartbeater.cc:464] Registering TS with master...
16:16:07.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.281603 24340 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:07.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.282853 24342 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:07.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.283223 24342 heartbeater.cc:464] Registering TS with master...
16:16:07.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.284207 24342 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:07.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.287402 22880 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:07.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.289072 22790 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:07.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.291247 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.67:54369
16:16:07.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.311967 24343 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:07.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.312466 24343 heartbeater.cc:464] Registering TS with master...
16:16:07.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.313376 24343 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:07.318 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] handling channelInactive
16:16:07.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:07.317431 22702 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:07.318 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state READY due to: connection closed
16:16:07.318 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:07.319 [DEBUG - Time-limited test] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:07.321 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.322 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.322 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.339 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:07.339 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:07.339 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@401960699(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@3af25b4e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48bad6c0 -> wakeup thread Time-limited test)) after lookup
16:16:07.340 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:07.341 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.341 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.341 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.359 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:07.359 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:07.359 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [21ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [22ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [22ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@401960699(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@3af25b4e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48bad6c0 -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284))) after lookup
16:16:07.360 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:07.362 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.362 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.363 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.379 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:07.380 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:07.380 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=61), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [21ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [22ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [22ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [44ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [60ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@401960699(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@3af25b4e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48bad6c0 -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598))) after lookup
16:16:07.381 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:07.382 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.383 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.383 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.499 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:07.500 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:07.500 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [21ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [22ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [22ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [44ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [60ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [61ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [64ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [64ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [180ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@401960699(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@3af25b4e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598) -> (continuation of Deferred@893136654 after retry RPC@1535830170), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48bad6c0 -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598) -> (continuation of Deferred@893136654 after retry RPC@1535830170))) after lookup
16:16:07.502 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:07.505 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.505 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.505 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.858 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:07.859 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:07.859 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=540), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [21ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [22ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [22ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [44ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [60ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [61ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [64ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [64ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [180ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [183ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [186ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [186ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [535ms] refreshing cache from master, [535ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [539ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@401960699(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@3af25b4e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598) -> (continuation of Deferred@893136654 after retry RPC@1535830170) -> (continuation of Deferred@82380357 after retry RPC@145485946), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48bad6c0 -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598) -> (continuation of Deferred@893136654 after retry RPC@1535830170) -> (continuation of Deferred@82380357 after retry RPC@145485946))) after lookup
16:16:07.860 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:07.861 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.861 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:07.861 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:08.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.297292 24340 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:08.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.520941 24354 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3de21ca168134ac9a117e9e6cfc3a740)
16:16:08.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.521739 24354 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:08.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.524322 24354 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:08.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:08.527617 24014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:08.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:08.531447 24014 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:08.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.539383 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "f679e1cd79344040853de5b0fe6e1b38" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" is_pre_election: true
16:16:08.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.540134 24291 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f679e1cd79344040853de5b0fe6e1b38 in term 4.
16:16:08.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.541584 24012 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38; no voters: 3de21ca168134ac9a117e9e6cfc3a740
16:16:08.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.542555 24354 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 5 FOLLOWER]: Leader pre-election won for term 6
16:16:08.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.542928 24354 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3de21ca168134ac9a117e9e6cfc3a740)
16:16:08.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.543306 24354 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 5 FOLLOWER]: Advancing to term 6
16:16:08.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.548388 24354 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:08.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.550022 24354 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 6 election: Requested vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:08.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.551038 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "f679e1cd79344040853de5b0fe6e1b38" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:16:08.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.551569 24291 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 4 FOLLOWER]: Advancing to term 6
16:16:08.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:08.553575 24014 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:08.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.559257 24291 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f679e1cd79344040853de5b0fe6e1b38 in term 6.
16:16:08.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.560325 24012 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38; no voters: 3de21ca168134ac9a117e9e6cfc3a740
16:16:08.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.561040 24354 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 6 FOLLOWER]: Leader election won for term 6
16:16:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.562876 24354 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 6 LEADER]: Becoming Leader. State: Replica: f679e1cd79344040853de5b0fe6e1b38, State: Running, Role: LEADER
16:16:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.564059 24354 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:08.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:08.573448 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 reported cstate change: term changed from 5 to 6, leader changed from 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65) to f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66). New cstate: current_term: 6 leader_uuid: "f679e1cd79344040853de5b0fe6e1b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: UNKNOWN } } }
16:16:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:09.054571 24291 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Refusing update from remote peer f679e1cd79344040853de5b0fe6e1b38: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
16:16:09.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:09.056231 24354 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
16:16:09.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:09.111029 24014 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 -> Peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Couldn't send request to peer 3de21ca168134ac9a117e9e6cfc3a740. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:09.559 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:09.559 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:09.560 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2240), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [21ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [22ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [22ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [41ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [43ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [44ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [60ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [61ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [64ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [64ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [180ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [183ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [186ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [186ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [535ms] refreshing cache from master, [535ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [539ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [541ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [542ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [542ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [2235ms] refreshing cache from master, [2235ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [2239ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@401960699(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@3af25b4e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598) -> (continuation of Deferred@893136654 after retry RPC@1535830170) -> (continuation of Deferred@82380357 after retry RPC@145485946) -> (continuation of Deferred@649867744 after retry RPC@1894346160), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48bad6c0 -> wakeup thread Time-limited test -> (continuation of Deferred@1927911768 after retry RPC@1695603284) -> (continuation of Deferred@1399935558 after retry RPC@810606598) -> (continuation of Deferred@893136654 after retry RPC@1535830170) -> (continuation of Deferred@82380357 after retry RPC@145485946) -> (continuation of Deferred@649867744 after retry RPC@1894346160))) after lookup
16:16:09.560 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:09.563 [DEBUG - kudu-nio-1] (Connection.java:757) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] Successfully connected to peer
16:16:09.589 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x029bd89f, L:/127.0.0.1:36428 - R:/127.22.31.66:39515] using TOKEN/null
16:16:09.647 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.65:36225
16:16:09.648 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:09.650030 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:09.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:36225
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40275
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:09.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:10.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.059019 24399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:10.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.059660 24399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:10.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.060330 24399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:10.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.097378 24399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:10.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.099624 24399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.65
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:10.143194 24399 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:36225
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40275
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:10.144845 24399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:10.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:10.147084 24399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:10.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.168668 24407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:11.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:11.355413 24014 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 -> Peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Couldn't send request to peer 3de21ca168134ac9a117e9e6cfc3a740. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:16:11.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.168751 24408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:11.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:10.171272 24399 server_base.cc:1034] running on GCE node
16:16:11.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:10.168798 24410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:11.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.430316 24399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:11.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.433557 24399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:11.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.435173 24399 hybrid_clock.cc:648] HybridClock initialized: now 1786464971435121 us; error 54 us; skew 500 ppm
16:16:11.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.436383 24399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:11.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.446442 24399 webserver.cc:460] Webserver started at http://127.22.31.65:40275/ using document root <none> and password file <none>
16:16:11.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.447582 24399 fs_manager.cc:359] Metadata directory not provided
16:16:11.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.447872 24399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:11.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.457998 24399 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.005s	sys 0.001s
16:16:11.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.464561 24419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:11.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.466075 24399 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
16:16:11.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.466590 24399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:11.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:16:11.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:40 on dist-test-slave-9jjr"
16:16:11.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.469944 24399 fs_report.cc:383] FS layout report
16:16:11.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:11.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/data
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:11.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:11.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.533164 24399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.535212 24399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.535697 24399 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:11.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.538786 24399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:11.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.548229 24427 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:11.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.557308 24399 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:11.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.557624 24399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
16:16:11.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.557870 24399 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:11.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.565220 24399 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:11.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.565526 24399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s	user 0.005s	sys 0.000s
16:16:11.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.565949 24427 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap starting.
16:16:11.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.660725 24427 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Log is configured to *not* fsync() on all Append() calls
16:16:11.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.753412 24427 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:11.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.754820 24427 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap complete.
16:16:11.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.757197 24427 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent bootstrapping tablet: real 0.192s	user 0.147s	sys 0.036s
16:16:11.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.780519 24399 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.65:36225
16:16:11.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.780709 24548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.65:36225 every 8 connection(s)
16:16:11.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.784866 24399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:16:11.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.795383 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 24399
16:16:11.797 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:11.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.791359 24427 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:11.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.798092 24427 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Initialized, Role: FOLLOWER
16:16:11.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.799602 24427 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:11.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.839814 24427 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent starting tablet: real 0.082s	user 0.040s	sys 0.017s
16:16:11.848 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:11.848 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.66:39515
16:16:11.849 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:16:11.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.849649 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 23990
16:16:11.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.856938 24549 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:11.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.857537 24549 heartbeater.cc:464] Registering TS with master...
16:16:11.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.858891 24550 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:11.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.859004 24549 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:11.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.859508 24550 heartbeater.cc:464] Registering TS with master...
16:16:11.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.860462 24550 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:11.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.864154 22878 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:11.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.864487 22790 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:11.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.868526 22878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.65:37309
16:16:11.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.876485 24551 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.876682 24549 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.876902 24551 heartbeater.cc:464] Registering TS with master...
16:16:11.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.877650 24551 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:11.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:11.883337 22702 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:11.898 [DEBUG - kudu-nio-1] (Connection.java:239) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] handling channelInactive
16:16:11.898 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state READY due to: connection closed
16:16:11.899 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:11.900 [DEBUG - Time-limited test] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:11.901 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.901 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.902 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.919 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:11.919 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:11.919 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [3ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@350902897(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@a35e824 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73b0ed93 -> wakeup thread Time-limited test)) after lookup
16:16:11.920 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:11.921 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.922 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.922 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.938 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:11.938 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:11.939 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [3ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [20ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [23ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [23ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@350902897(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@a35e824 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73b0ed93 -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812))) after lookup
16:16:11.939 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:11.940 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.941 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.941 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.980 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:11.980 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:11.980 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [3ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [20ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [23ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [23ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [42ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [42ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [80ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@350902897(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@a35e824 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73b0ed93 -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264))) after lookup
16:16:11.981 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:11.983 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.983 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:11.984 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:12.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:12.039 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:12.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=140), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [3ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [20ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [23ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [23ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [42ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [42ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [80ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [82ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [85ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [86ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [139ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@350902897(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@a35e824 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264) -> (continuation of Deferred@2098913193 after retry RPC@284480362), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73b0ed93 -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264) -> (continuation of Deferred@2098913193 after retry RPC@284480362))) after lookup
16:16:12.040 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:12.041 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:12.041 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:12.041 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:12.359 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:12.359 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:12.359 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=460), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [3ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [20ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [23ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [23ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [42ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [42ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [80ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [82ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [85ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [86ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [139ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [141ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [142ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [142ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [455ms] refreshing cache from master, [456ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [459ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@350902897(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@a35e824 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264) -> (continuation of Deferred@2098913193 after retry RPC@284480362) -> (continuation of Deferred@322930044 after retry RPC@496226811), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73b0ed93 -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264) -> (continuation of Deferred@2098913193 after retry RPC@284480362) -> (continuation of Deferred@322930044 after retry RPC@496226811))) after lookup
16:16:12.360 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:12.362 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:12.362 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:12.362 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:13.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.212003 24561 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:16:13.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.212705 24561 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:13.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.222000 24561 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:13.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:13.225780 24420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:13.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.225769 24565 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f679e1cd79344040853de5b0fe6e1b38)
16:16:13.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.226531 24565 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:13.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.230903 24565 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:13.244982 24420 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:13.247954 24213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:13.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.255080 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" is_pre_election: true
16:16:13.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:13.255162 24213 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:13.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.255975 24291 raft_consensus.cc:2379] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3de21ca168134ac9a117e9e6cfc3a740 in current term 6: Already voted for candidate f679e1cd79344040853de5b0fe6e1b38 in this term.
16:16:13.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.257298 24421 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740; no voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38
16:16:13.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.258368 24561 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 5 FOLLOWER]: Advancing to term 6
16:16:13.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.263052 24498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740" is_pre_election: true
16:16:13.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.266566 24561 raft_consensus.cc:2726] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
16:16:13.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.267226 24498 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 6.
16:16:13.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.272917 24216 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: f679e1cd79344040853de5b0fe6e1b38
16:16:13.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.273700 24565 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Leader pre-election won for term 7
16:16:13.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.274056 24565 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Starting leader election (detected failure of leader f679e1cd79344040853de5b0fe6e1b38)
16:16:13.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.274374 24565 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 6 FOLLOWER]: Advancing to term 7
16:16:13.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.279023 24565 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:13.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.280879 24565 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 7 election: Requested vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:13.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.282200 24498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:16:13.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.282866 24498 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 6 FOLLOWER]: Advancing to term 7
16:16:13.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:13.284301 24213 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:13.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.288303 24498 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 7.
16:16:13.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.289482 24216 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: f679e1cd79344040853de5b0fe6e1b38
16:16:13.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.290367 24565 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Leader election won for term 7
16:16:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.292271 24565 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 LEADER]: Becoming Leader. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Running, Role: LEADER
16:16:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.293296 24565 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:13.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.302973 22878 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa reported cstate change: term changed from 6 to 7, leader changed from f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66) to bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67). New cstate: current_term: 7 leader_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: UNKNOWN } } }
16:16:13.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.790865 24498 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 7 FOLLOWER]: Refusing update from remote peer bdcfad5657c742588c55c4e90e2ac7fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
16:16:13.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.792651 24565 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
16:16:13.803 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:13.803 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:13.805 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1904), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [3ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [3ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [20ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [23ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [23ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [39ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [40ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [42ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [42ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [80ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [82ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [85ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [86ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [139ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [141ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [142ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [142ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [455ms] refreshing cache from master, [456ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [459ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [461ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [463ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [463ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [1895ms] refreshing cache from master, [1896ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1902ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@350902897(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@a35e824 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264) -> (continuation of Deferred@2098913193 after retry RPC@284480362) -> (continuation of Deferred@322930044 after retry RPC@496226811) -> (continuation of Deferred@188512198 after retry RPC@136536915), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@73b0ed93 -> wakeup thread Time-limited test -> (continuation of Deferred@699809724 after retry RPC@1562280812) -> (continuation of Deferred@1964376054 after retry RPC@1770033264) -> (continuation of Deferred@2098913193 after retry RPC@284480362) -> (continuation of Deferred@322930044 after retry RPC@496226811) -> (continuation of Deferred@188512198 after retry RPC@136536915))) after lookup
16:16:13.810 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:13.814 [DEBUG - kudu-nio-0] (Connection.java:757) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] Successfully connected to peer
16:16:13.870 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x674a6733, L:/127.0.0.1:36854 - R:/127.22.31.67:33649] using TOKEN/null
16:16:13.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:13.871397 24213 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa -> Peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Couldn't send request to peer f679e1cd79344040853de5b0fe6e1b38. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:13.927 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.66:39515
16:16:13.927 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:13.929075 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:13.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:39515
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36227
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:13.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:14.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.333155 24597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:14.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.333679 24597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:14.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.334365 24597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:14.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.368640 24597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:14.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.369509 24597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.66
16:16:14.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:14.408362 24597 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:14.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:14.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:14.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:39515
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36227
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:14.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:14.409636 24597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:14.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:14.411309 24597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:14.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.433241 24604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.433241 24603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:14.434919 24607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:15.708966 24606 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:16:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.709173 24597 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:16:15.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.714864 24597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:15.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.718065 24597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:15.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.719640 24597 hybrid_clock.cc:648] HybridClock initialized: now 1786464975719567 us; error 89 us; skew 500 ppm
16:16:15.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.720494 24597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:15.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.739924 24597 webserver.cc:460] Webserver started at http://127.22.31.66:36227/ using document root <none> and password file <none>
16:16:15.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.741130 24597 fs_manager.cc:359] Metadata directory not provided
16:16:15.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.741367 24597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:15.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.751047 24597 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.001s
16:16:15.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.756250 24618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:15.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.757588 24597 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
16:16:15.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.757894 24597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:43 on dist-test-slave-9jjr"
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.760578 24597 fs_report.cc:383] FS layout report
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/data
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:15.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:15.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.822078 24597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:15.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.823943 24597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:15.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.824513 24597 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:15.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.827356 24597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:15.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.834846 24626 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:15.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.843228 24597 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:15.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.843545 24597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.003s
16:16:15.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.843887 24597 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:15.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.850601 24597 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:15.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.850883 24597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s	user 0.006s	sys 0.001s
16:16:15.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.851181 24626 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap starting.
16:16:15.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:15.937760 24626 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Log is configured to *not* fsync() on all Append() calls
16:16:16.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.039016 24626 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:16.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.040473 24626 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap complete.
16:16:16.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.042624 24626 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent bootstrapping tablet: real 0.192s	user 0.129s	sys 0.054s
16:16:16.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.075662 24626 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:16.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.079324 24626 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: f679e1cd79344040853de5b0fe6e1b38, State: Initialized, Role: FOLLOWER
16:16:16.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.080806 24626 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:16.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.086014 24626 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent starting tablet: real 0.043s	user 0.040s	sys 0.000s
16:16:16.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.136962 24597 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.66:39515
16:16:16.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.137173 24745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.66:39515 every 8 connection(s)
16:16:16.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.139998 24597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:16.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.151783 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 24597
16:16:16.154 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:16.190 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:16.191 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.67:33649
16:16:16.191 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:16:16.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.191514 24746 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:16.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.191897 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 24183
16:16:16.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.192022 24746 heartbeater.cc:464] Registering TS with master...
16:16:16.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.193365 24746 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:16.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.196055 24748 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:16.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.196403 24748 heartbeater.cc:464] Registering TS with master...
16:16:16.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.197286 24748 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:16.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.198278 22880 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:16.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.201409 22701 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:16.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.201658 24747 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:16.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.202010 24747 heartbeater.cc:464] Registering TS with master...
16:16:16.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.202225 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.66:43227
16:16:16.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.203205 24747 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:16.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:16.207031 22790 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:16.233 [DEBUG - kudu-nio-0] (Connection.java:239) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] handling channelInactive
16:16:16.233 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state READY due to: connection closed
16:16:16.234 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:16.234 [DEBUG - Time-limited test] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.236 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.236 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.236 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.259 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:16.259 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:16.260 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test)) after lookup
16:16:16.260 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.261 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.261 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.262 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.279 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:16.279 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:16.279 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [44ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312))) after lookup
16:16:16.280 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.282 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.282 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.282 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.299 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:16.299 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:16.300 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=65), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [44ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [46ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [48ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [65ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838))) after lookup
16:16:16.300 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.301 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.302 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.302 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.398 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:16.399 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:16.399 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=164), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [44ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [46ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [48ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [65ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [66ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [68ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [68ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [160ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [163ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792))) after lookup
16:16:16.400 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.402 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.402 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.402 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.518 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:16.519 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:16.519 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=285), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [44ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [46ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [48ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [65ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [66ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [68ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [68ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [160ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [163ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [166ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [168ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [168ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [280ms] refreshing cache from master, [280ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [284ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318))) after lookup
16:16:16.520 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.521 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.522 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.522 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.699 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:16.699 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:16.699 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=465), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [44ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [46ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [48ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [65ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [66ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [68ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [68ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [160ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [163ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [166ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [168ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [168ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [280ms] refreshing cache from master, [280ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [284ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [286ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [288ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [288ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [460ms] refreshing cache from master, [460ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [464ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318) -> (continuation of Deferred@96614116 after retry RPC@1359261602), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318) -> (continuation of Deferred@96614116 after retry RPC@1359261602))) after lookup
16:16:16.700 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.702 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.702 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.702 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.838 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:16.838 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:16.839 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=604), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [44ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [46ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [48ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [65ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [66ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [68ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [68ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [160ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [163ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [166ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [168ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [168ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [280ms] refreshing cache from master, [280ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [284ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [286ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [288ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [288ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [460ms] refreshing cache from master, [460ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [464ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [466ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [468ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [468ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [600ms] refreshing cache from master, [600ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [604ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318) -> (continuation of Deferred@96614116 after retry RPC@1359261602) -> (continuation of Deferred@411649375 after retry RPC@1820622931), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318) -> (continuation of Deferred@96614116 after retry RPC@1359261602) -> (continuation of Deferred@411649375 after retry RPC@1820622931))) after lookup
16:16:16.840 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:16.841 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.841 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:16.841 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:17.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.207647 24746 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:17.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.494246 24758 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:16:17.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.494704 24758 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:17.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.496315 24758 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:17.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:17.504386 24421 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:16:17.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.511981 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38" is_pre_election: true
16:16:17.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.512745 24694 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 6.
16:16:17.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.514122 24420 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:16:17.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.515024 24758 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 7 FOLLOWER]: Leader pre-election won for term 8
16:16:17.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.515398 24758 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 7 FOLLOWER]: Starting leader election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:16:17.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.515657 24758 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 7 FOLLOWER]: Advancing to term 8
16:16:17.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.520102 24758 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:17.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.521754 24758 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 8 election: Requested vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:17.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.522993 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:16:17.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.523681 24694 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 6 FOLLOWER]: Advancing to term 8
16:16:17.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:17.525578 24421 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:16:17.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.531908 24694 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 8.
16:16:17.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.533011 24420 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:16:17.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.533847 24758 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Leader election won for term 8
16:16:17.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.535161 24758 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 LEADER]: Becoming Leader. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Running, Role: LEADER
16:16:17.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.536125 24758 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:17.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:17.545435 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 reported cstate change: term changed from 7 to 8, leader changed from bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67) to 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65). New cstate: current_term: 8 leader_uuid: "3de21ca168134ac9a117e9e6cfc3a740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: HEALTHY } } }
16:16:17.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:17.991076 24421 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 -> Peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Couldn't send request to peer bdcfad5657c742588c55c4e90e2ac7fa. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:18.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:18.141716 24694 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 8 FOLLOWER]: Refusing update from remote peer 3de21ca168134ac9a117e9e6cfc3a740: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
16:16:18.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:18.144253 24758 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
16:16:18.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:18.455577 24421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111) [suppressed 3 similar messages]
16:16:18.879 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:18.879 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:18.880 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=16, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2645), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [25ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [26ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [28ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [28ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [44ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [46ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [48ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [65ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [66ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [68ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [68ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [160ms] refreshing cache from master, [160ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [163ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [166ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [168ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [168ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [280ms] refreshing cache from master, [280ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [284ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [286ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [288ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [288ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [460ms] refreshing cache from master, [460ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [464ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [466ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [468ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [468ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [600ms] refreshing cache from master, [600ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [604ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [605ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [607ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [607ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2641ms] refreshing cache from master, [2641ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [2645ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1951497935(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@377dc061 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318) -> (continuation of Deferred@96614116 after retry RPC@1359261602) -> (continuation of Deferred@411649375 after retry RPC@1820622931) -> (continuation of Deferred@947477699 after retry RPC@932333449), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5241b229 -> wakeup thread Time-limited test -> (continuation of Deferred@2048262914 after retry RPC@2088991312) -> (continuation of Deferred@279042392 after retry RPC@1998020838) -> (continuation of Deferred@125395109 after retry RPC@1987711792) -> (continuation of Deferred@339526754 after retry RPC@846837318) -> (continuation of Deferred@96614116 after retry RPC@1359261602) -> (continuation of Deferred@411649375 after retry RPC@1820622931) -> (continuation of Deferred@947477699 after retry RPC@932333449))) after lookup
16:16:18.881 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:18.883 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] Successfully connected to peer
16:16:18.911 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x45db6cb5, L:/127.0.0.1:34466 - R:/127.22.31.65:36225] using TOKEN/null
16:16:18.971 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.67:33649
16:16:18.972 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:18.973735 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:18.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:18.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:19.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.339854 24796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:19.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.340600 24796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:19.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.341259 24796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:19.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.376523 24796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:19.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.377557 24796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.67
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:19.416774 24796 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:19.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:19.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:19.418445 24796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:19.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:19.420347 24796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.444355 24803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:20.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:20.509008 24421 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 -> Peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Couldn't send request to peer bdcfad5657c742588c55c4e90e2ac7fa. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:16:20.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.444586 24802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:20.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:19.444386 24805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:20.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:20.804757 24804 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1361 milliseconds
16:16:20.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.805047 24796 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:16:20.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.811211 24796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:20.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.813851 24796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:20.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.815572 24796 hybrid_clock.cc:648] HybridClock initialized: now 1786464980815520 us; error 64 us; skew 500 ppm
16:16:20.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.817271 24796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:20.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.834550 24796 webserver.cc:460] Webserver started at http://127.22.31.67:36945/ using document root <none> and password file <none>
16:16:20.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.835690 24796 fs_manager.cc:359] Metadata directory not provided
16:16:20.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.835948 24796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:20.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.846570 24796 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.001s
16:16:20.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.851716 24822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.853080 24796 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
16:16:20.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.853531 24796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:20.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:16:20.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:46 on dist-test-slave-9jjr"
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.856267 24796 fs_report.cc:383] FS layout report
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/data
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:20.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:20.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.915300 24796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:20.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.917028 24796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:20.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.917460 24796 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:20.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.920199 24796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:20.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.926393 24829 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:20.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.935472 24796 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:20.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.935820 24796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
16:16:20.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.936432 24796 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.942960 24796 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.943374 24796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s	user 0.007s	sys 0.000s
16:16:20.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:20.943630 24829 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap starting.
16:16:21.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.037195 24829 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Log is configured to *not* fsync() on all Append() calls
16:16:21.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.165031 24829 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:21.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.166553 24829 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap complete.
16:16:21.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.169107 24829 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent bootstrapping tablet: real 0.226s	user 0.174s	sys 0.033s
16:16:21.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.200063 24829 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:21.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.204772 24829 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Initialized, Role: FOLLOWER
16:16:21.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.205932 24829 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:21.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.224061 24829 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent starting tablet: real 0.054s	user 0.045s	sys 0.007s
16:16:21.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.273917 24796 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.67:33649
16:16:21.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.274034 24953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.67:33649 every 8 connection(s)
16:16:21.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.277490 24796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:21.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.282331 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 24796
16:16:21.283 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:21.321 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:21.323 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.65:36225
16:16:21.323 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:16:21.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.323982 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 24399
16:16:21.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.350426 24955 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:21.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.350927 24955 heartbeater.cc:464] Registering TS with master...
16:16:21.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.352285 24955 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:21.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.357067 24954 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:21.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.357482 24954 heartbeater.cc:464] Registering TS with master...
16:16:21.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.357450 22790 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:21.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.358469 24954 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:21.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.361862 22880 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:21.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.362082 24956 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:21.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.362672 24956 heartbeater.cc:464] Registering TS with master...
16:16:21.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.363644 24956 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:21.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.365255 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.67:47961
16:16:21.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:21.367426 22702 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:21.385 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] handling channelInactive
16:16:21.385 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state READY due to: connection closed
16:16:21.385 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:21.386 [DEBUG - Time-limited test] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:21.388 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.388 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.388 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.398 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:21.398 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:21.398 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@795014606(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1fd15b90 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33446b05 -> wakeup thread Time-limited test)) after lookup
16:16:21.399 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:21.401 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.401 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.401 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.419 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:21.420 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:21.420 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [16ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [34ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@795014606(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1fd15b90 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33446b05 -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192))) after lookup
16:16:21.422 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:21.423 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.423 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.423 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.459 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:21.460 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:21.460 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [16ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [34ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [36ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [38ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [74ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@795014606(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1fd15b90 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33446b05 -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106))) after lookup
16:16:21.461 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:21.463 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.463 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.463 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.580 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:21.580 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:21.580 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=195), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [16ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [34ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [36ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [38ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [74ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [75ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [78ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [189ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [194ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@795014606(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1fd15b90 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106) -> (continuation of Deferred@1721805978 after retry RPC@1144696273), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33446b05 -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106) -> (continuation of Deferred@1721805978 after retry RPC@1144696273))) after lookup
16:16:21.582 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:21.583 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.583 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.584 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.718 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:21.719 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:21.719 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=333), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [16ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [34ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [36ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [38ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [74ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [75ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [78ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [189ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [194ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [196ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [199ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [199ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [329ms] refreshing cache from master, [329ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [333ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@795014606(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1fd15b90 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106) -> (continuation of Deferred@1721805978 after retry RPC@1144696273) -> (continuation of Deferred@952004177 after retry RPC@1506273275), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33446b05 -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106) -> (continuation of Deferred@1721805978 after retry RPC@1144696273) -> (continuation of Deferred@952004177 after retry RPC@1506273275))) after lookup
16:16:21.720 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:21.722 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.722 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state CONNECTING due to: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:21.723 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225
16:16:22.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.370548 24954 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:22.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.509626 24966 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:16:22.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.510114 24966 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:22.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.512485 24966 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:22.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:22.517838 24826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:22.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:22.522363 24826 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.528539 24694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38" is_pre_election: true
16:16:22.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.530740 24823 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bdcfad5657c742588c55c4e90e2ac7fa; no voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38
16:16:22.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.532198 24966 raft_consensus.cc:2726] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
16:16:22.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.546677 24972 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3de21ca168134ac9a117e9e6cfc3a740)
16:16:22.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.547148 24972 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:22.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.549685 24972 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:22.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:22.553663 24623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:22.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:22.558571 24623 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:22.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.565850 24902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "f679e1cd79344040853de5b0fe6e1b38" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" is_pre_election: true
16:16:22.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.566758 24902 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f679e1cd79344040853de5b0fe6e1b38 in term 7.
16:16:22.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.568235 24621 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38; no voters: 3de21ca168134ac9a117e9e6cfc3a740
16:16:22.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.568960 24972 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 8 FOLLOWER]: Leader pre-election won for term 9
16:16:22.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.569293 24972 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3de21ca168134ac9a117e9e6cfc3a740)
16:16:22.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.569609 24972 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 8 FOLLOWER]: Advancing to term 9
16:16:22.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.574594 24972 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:22.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.576256 24972 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 9 election: Requested vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:22.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.577512 24902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "f679e1cd79344040853de5b0fe6e1b38" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:16:22.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.578197 24902 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 7 FOLLOWER]: Advancing to term 9
16:16:22.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:22.580240 24623 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111)
16:16:22.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.586372 24902 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f679e1cd79344040853de5b0fe6e1b38 in term 9.
16:16:22.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.587473 24621 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38; no voters: 3de21ca168134ac9a117e9e6cfc3a740
16:16:22.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.588220 24972 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 9 FOLLOWER]: Leader election won for term 9
16:16:22.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.589963 24972 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 9 LEADER]: Becoming Leader. State: Replica: f679e1cd79344040853de5b0fe6e1b38, State: Running, Role: LEADER
16:16:22.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.590855 24972 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:22.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:22.600147 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 reported cstate change: term changed from 8 to 9, leader changed from 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65) to f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66). New cstate: current_term: 9 leader_uuid: "f679e1cd79344040853de5b0fe6e1b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: UNKNOWN } } }
16:16:23.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:23.150101 24902 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 9 FOLLOWER]: Refusing update from remote peer f679e1cd79344040853de5b0fe6e1b38: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
16:16:23.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:23.152482 24991 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.001s
16:16:23.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:23.201581 24623 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 -> Peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Couldn't send request to peer 3de21ca168134ac9a117e9e6cfc3a740. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:23.679 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:23.679 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:23.680 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2294), Traces: [0ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [3ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [3ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [16ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [16ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [34ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [36ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [38ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [38ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [70ms] refreshing cache from master, [70ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [74ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [75ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [78ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [78ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [189ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [194ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [196ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [199ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [199ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [329ms] refreshing cache from master, [329ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [333ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [335ms] sending RPC to server 3de21ca168134ac9a117e9e6cfc3a740, [338ms] received response from server 3de21ca168134ac9a117e9e6cfc3a740: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [338ms] delaying RPC due to: Network error: Failed to connect to peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225): Connection refused: /127.22.31.65:36225, [2289ms] refreshing cache from master, [2290ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [2293ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@795014606(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1fd15b90 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106) -> (continuation of Deferred@1721805978 after retry RPC@1144696273) -> (continuation of Deferred@952004177 after retry RPC@1506273275) -> (continuation of Deferred@679264435 after retry RPC@1323434616), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33446b05 -> wakeup thread Time-limited test -> (continuation of Deferred@1253777049 after retry RPC@1521965192) -> (continuation of Deferred@1270840081 after retry RPC@764536106) -> (continuation of Deferred@1721805978 after retry RPC@1144696273) -> (continuation of Deferred@952004177 after retry RPC@1506273275) -> (continuation of Deferred@679264435 after retry RPC@1323434616))) after lookup
16:16:23.681 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:23.682 [DEBUG - kudu-nio-0] (Connection.java:757) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] Successfully connected to peer
16:16:23.703 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xfd1cc2e3, L:/127.0.0.1:56556 - R:/127.22.31.66:39515] using TOKEN/null
16:16:23.756 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.65:36225
16:16:23.756 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:23.758266 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:16:23.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:36225
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40275
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:24.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.150496 25011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:24.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.151015 25011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:24.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.151476 25011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:24.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.190774 25011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:24.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.191655 25011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.65
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:24.232201 25011 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.65:36225
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.65
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40275
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.65
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/logs
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:24.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:24.233554 25011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:24.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:24.235538 25011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:24.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.259660 25020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:25.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.259667 25021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:25.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:24.262287 25011 server_base.cc:1034] running on GCE node
16:16:25.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:24.260655 25023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:25.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.534621 25011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:25.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.537263 25011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:25.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.538756 25011 hybrid_clock.cc:648] HybridClock initialized: now 1786464985538581 us; error 185 us; skew 500 ppm
16:16:25.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.539716 25011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:25.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.546599 25011 webserver.cc:460] Webserver started at http://127.22.31.65:40275/ using document root <none> and password file <none>
16:16:25.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.547559 25011 fs_manager.cc:359] Metadata directory not provided
16:16:25.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.547756 25011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:25.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.556461 25011 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.000s
16:16:25.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.561869 25035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:25.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.563651 25011 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
16:16:25.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:25.565876 24623 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 -> Peer 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225): Couldn't send request to peer 3de21ca168134ac9a117e9e6cfc3a740. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.65:36225: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:16:25.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.564073 25011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:40 on dist-test-slave-9jjr"
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.566989 25011 fs_report.cc:383] FS layout report
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/wal
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/data
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:25.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:25.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:25.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:25.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.619946 25011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:25.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.622089 25011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:25.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.622885 25011 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:25.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.626050 25011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:25.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.634291 25044 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:25.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.644129 25011 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:25.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.644511 25011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s	user 0.002s	sys 0.001s
16:16:25.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.644866 25011 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:25.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.652387 25011 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:25.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.652750 25011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s	user 0.003s	sys 0.004s
16:16:25.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.653162 25044 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap starting.
16:16:25.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.744094 25044 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Log is configured to *not* fsync() on all Append() calls
16:16:25.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.853132 25011 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.65:36225
16:16:25.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.853560 25162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.65:36225 every 8 connection(s)
16:16:25.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.857530 25011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-0/data/info.pb
16:16:25.866 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:25.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.859709 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 25011
16:16:25.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.894661 25044 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:25.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.902495 25044 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Bootstrap complete.
16:16:25.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.904752 25044 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent bootstrapping tablet: real 0.252s	user 0.187s	sys 0.056s
16:16:25.909 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:25.910 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.66:39515
16:16:25.910 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:16:25.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.911288 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 24597
16:16:25.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.945188 25044 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:25.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.953145 25044 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Initialized, Role: FOLLOWER
16:16:25.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.954416 25044 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:25.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.969801 25165 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:25.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.970206 25044 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740: Time spent starting tablet: real 0.065s	user 0.039s	sys 0.012s
16:16:25.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.970520 25165 heartbeater.cc:464] Registering TS with master...
16:16:25.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.971972 25165 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:25.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.978595 22789 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:25.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.980094 25163 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:25.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.980489 25163 heartbeater.cc:464] Registering TS with master...
16:16:25.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.981393 25163 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:25.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.981978 25170 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:25.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.982371 25170 heartbeater.cc:464] Registering TS with master...
16:16:25.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.983368 25170 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:25.990 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:25.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:25.986338 22878 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:25.994 [DEBUG - kudu-nio-0] (Connection.java:239) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] handling channelInactive
16:16:25.995 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state READY due to: connection closed
16:16:25.995 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: connection closed
16:16:26.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:26.000149 22878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.65:38539
16:16:26.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:26.001760 22701 ts_manager.cc:194] Re-registered known tserver with Master: 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:26.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:26.007186 25163 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:26.021 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:26.021 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:26.022 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [4ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [29ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1172339072(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1065d506 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a4f838a -> wakeup thread Time-limited test)) after lookup
16:16:26.023 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:26.029 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.029 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.029 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.040 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:26.041 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:26.041 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [4ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [29ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [31ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [38ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [38ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [48ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1172339072(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1065d506 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a4f838a -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197))) after lookup
16:16:26.043 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:26.044 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.044 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.044 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.060 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:26.060 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:26.060 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [4ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [29ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [31ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [38ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [38ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [48ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [51ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [53ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [53ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1172339072(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1065d506 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a4f838a -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702))) after lookup
16:16:26.061 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:26.063 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.063 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.063 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.199 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:26.199 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:26.200 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=208), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [4ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [29ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [31ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [38ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [38ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [48ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [51ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [53ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [53ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [70ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [72ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [72ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [203ms] refreshing cache from master, [203ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [207ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1172339072(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1065d506 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702) -> (continuation of Deferred@1860270832 after retry RPC@1114037053), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a4f838a -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702) -> (continuation of Deferred@1860270832 after retry RPC@1114037053))) after lookup
16:16:26.201 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:26.202 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.202 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.202 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.479 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:26.479 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)[L]]}]
16:16:26.480 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=489), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [4ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [29ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [31ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [38ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [38ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [48ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [51ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [53ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [53ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [70ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [72ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [72ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [203ms] refreshing cache from master, [203ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [207ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [209ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [211ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [211ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [483ms] refreshing cache from master, [483ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [487ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1172339072(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1065d506 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@92262082 after retry RPC@631144652), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a4f838a -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@92262082 after retry RPC@631144652))) after lookup
16:16:26.480 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] connecting to peer
16:16:26.481 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.482 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] cleaning up while in state CONNECTING due to: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:26.482 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515
16:16:27.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.103955 25178 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:16:27.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.104429 25178 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:27.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.106771 25178 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:27.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:27.111552 25036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:27.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:27.115559 25036 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:27.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.121100 24902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" is_pre_election: true
16:16:27.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.122820 25038 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740; no voters: bdcfad5657c742588c55c4e90e2ac7fa, f679e1cd79344040853de5b0fe6e1b38
16:16:27.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.123633 25178 raft_consensus.cc:2726] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
16:16:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.276611 25184 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f679e1cd79344040853de5b0fe6e1b38)
16:16:27.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.277267 25184 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:27.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.279490 25184 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:27.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:27.292301 24823 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:27.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.301988 25116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740" is_pre_election: true
16:16:27.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.302760 25116 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 8.
16:16:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.304070 24826 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: f679e1cd79344040853de5b0fe6e1b38
16:16:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.304754 25184 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 9 FOLLOWER]: Leader pre-election won for term 10
16:16:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.305045 25184 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 9 FOLLOWER]: Starting leader election (detected failure of leader f679e1cd79344040853de5b0fe6e1b38)
16:16:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.305264 25184 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 9 FOLLOWER]: Advancing to term 10
16:16:27.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.310034 25184 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:27.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.311574 25184 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 10 election: Requested vote from peers f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515), 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65:36225)
16:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.312755 25116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "3de21ca168134ac9a117e9e6cfc3a740"
16:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.313371 25116 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 8 FOLLOWER]: Advancing to term 10
16:16:27.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:27.315049 24823 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111)
16:16:27.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.320734 25116 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcfad5657c742588c55c4e90e2ac7fa in term 10.
16:16:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.321771 24826 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, bdcfad5657c742588c55c4e90e2ac7fa; no voters: f679e1cd79344040853de5b0fe6e1b38
16:16:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.322559 25184 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 10 FOLLOWER]: Leader election won for term 10
16:16:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.323684 25184 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 10 LEADER]: Becoming Leader. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Running, Role: LEADER
16:16:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.324617 25184 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:27.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.334092 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa reported cstate change: term changed from 9 to 10, leader changed from f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66) to bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67). New cstate: current_term: 10 leader_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: UNKNOWN } } }
16:16:27.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:27.881778 24823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111) [suppressed 2 similar messages]
16:16:27.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.882099 25116 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 10 FOLLOWER]: Refusing update from remote peer bdcfad5657c742588c55c4e90e2ac7fa: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
16:16:27.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:27.884079 25202 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
16:16:27.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:27.885331 24823 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa -> Peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Couldn't send request to peer f679e1cd79344040853de5b0fe6e1b38. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:28.459 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:28.459 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:28.460 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2468), Traces: [0ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [4ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [29ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [31ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [38ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [38ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [48ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [51ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [53ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [53ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [63ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [70ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [72ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [72ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [203ms] refreshing cache from master, [203ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [207ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [209ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [211ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [211ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [483ms] refreshing cache from master, [483ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [487ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [489ms] sending RPC to server f679e1cd79344040853de5b0fe6e1b38, [491ms] received response from server f679e1cd79344040853de5b0fe6e1b38: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [491ms] delaying RPC due to: Network error: Failed to connect to peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515): Connection refused: /127.22.31.66:39515, [2463ms] refreshing cache from master, [2463ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [2467ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@1172339072(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@1065d506 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@92262082 after retry RPC@631144652) -> (continuation of Deferred@654619180 after retry RPC@1495181481), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a4f838a -> wakeup thread Time-limited test -> (continuation of Deferred@1232333410 after retry RPC@2140562197) -> (continuation of Deferred@1308431421 after retry RPC@362231702) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@92262082 after retry RPC@631144652) -> (continuation of Deferred@654619180 after retry RPC@1495181481))) after lookup
16:16:28.461 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:28.462 [DEBUG - kudu-nio-0] (Connection.java:757) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] Successfully connected to peer
16:16:28.485 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xef2ccca0, L:/127.0.0.1:41320 - R:/127.22.31.67:33649] using TOKEN/null
16:16:28.534 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.66:39515
16:16:28.535 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

16:16:28.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:28.536414 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:28.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:28.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:28.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:16:28.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:39515
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36227
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:28.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:28.893741 25224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:28.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:28.894608 25224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:28.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:28.895527 25224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:28.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:28.933939 25224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:28.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:28.935010 25224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.66
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:28.974970 25224 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.66:39515
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.66
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36227
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:28.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.66
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/logs
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:28.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:28.976286 25224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:28.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:28.978740 25224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:29.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:29.006052 25231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:30.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:29.007509 25232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:30.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:30.349603 25234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:30.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:30.351846 25224 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 1.345s	user 0.005s	sys 0.000s
16:16:30.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:30.352162 25224 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 1.345s	user 0.005s	sys 0.000s
16:16:30.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.352429 25224 server_base.cc:1034] running on GCE node
16:16:30.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.353684 25224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:30.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.356279 25224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:30.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.357875 25224 hybrid_clock.cc:648] HybridClock initialized: now 1786464990357844 us; error 53 us; skew 500 ppm
16:16:30.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.358973 25224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:30.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.368510 25224 webserver.cc:460] Webserver started at http://127.22.31.66:36227/ using document root <none> and password file <none>
16:16:30.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.369848 25224 fs_manager.cc:359] Metadata directory not provided
16:16:30.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.370136 25224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:30.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.380511 25224 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.003s	sys 0.004s
16:16:30.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.386569 25245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:30.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.388128 25224 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.001s
16:16:30.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.388540 25224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:30.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:16:30.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:43 on dist-test-slave-9jjr"
16:16:30.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.391366 25224 fs_report.cc:383] FS layout report
16:16:30.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/wal
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/data
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:30.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:30.423097 24823 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa -> Peer f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515): Couldn't send request to peer f679e1cd79344040853de5b0fe6e1b38. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.66:39515: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:16:30.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.451727 25224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:30.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.453488 25224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:30.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.453986 25224 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:30.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.457427 25224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:30.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.464632 25255 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:30.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.474211 25224 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:30.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.474539 25224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.000s
16:16:30.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.474762 25224 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:30.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.480100 25224 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:30.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.480315 25224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
16:16:30.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.480957 25255 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap starting.
16:16:30.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.572773 25255 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Log is configured to *not* fsync() on all Append() calls
16:16:30.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.709586 25224 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.66:39515
16:16:30.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.709949 25374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.66:39515 every 8 connection(s)
16:16:30.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.711035 25255 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:30.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.712018 25255 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Bootstrap complete.
16:16:30.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.713608 25255 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent bootstrapping tablet: real 0.233s	user 0.175s	sys 0.049s
16:16:30.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.713763 25224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-1/data/info.pb
16:16:30.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.723012 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 25224
16:16:30.724 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:30.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.757387 25255 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:30.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.765605 25255 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: f679e1cd79344040853de5b0fe6e1b38, State: Initialized, Role: FOLLOWER
16:16:30.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.767302 25255 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:30.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.783057 25376 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:30.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.783679 25376 heartbeater.cc:464] Registering TS with master...
16:16:30.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.784929 25255 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38: Time spent starting tablet: real 0.071s	user 0.039s	sys 0.018s
16:16:30.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.785180 25376 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:30.792 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:30.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.791185 22790 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:30.792 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.31.67:33649
16:16:30.793 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:16:30.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.793681 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 24796
16:16:30.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.805200 25377 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:30.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.805636 25377 heartbeater.cc:464] Registering TS with master...
16:16:30.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.807018 25377 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:30.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.811327 22701 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:30.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.811403 25375 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:30.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.811841 25375 heartbeater.cc:464] Registering TS with master...
16:16:30.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.812922 25375 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:30.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.817013 22880 ts_manager.cc:194] Re-registered known tserver with Master: f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:30.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.821300 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.66:44961
16:16:30.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:30.827618 25375 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:30.846 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:30.846 [DEBUG - kudu-nio-0] (Connection.java:239) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] handling channelInactive
16:16:30.847 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state READY due to: connection closed
16:16:30.847 [DEBUG - Time-limited test] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:30.849 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.849 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.849 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.860 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:30.860 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:30.860 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@422739586(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@32ccee3d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@83835a5 -> wakeup thread Time-limited test)) after lookup
16:16:30.862 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:30.863 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.863 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.863 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.879 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:30.880 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:30.881 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [16ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [16ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [32ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@422739586(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@32ccee3d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@83835a5 -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076))) after lookup
16:16:30.882 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:30.883 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.884 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.884 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.899 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:30.899 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:30.900 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [16ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [16ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [32ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [34ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [37ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [37ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [51ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@422739586(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@32ccee3d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@83835a5 -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910))) after lookup
16:16:30.901 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:30.901 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.902 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.902 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:30.999 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:30.999 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:31.000 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=152), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [16ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [16ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [32ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [34ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [37ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [37ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [51ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [53ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [55ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [55ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [151ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@422739586(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@32ccee3d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910) -> (continuation of Deferred@2110108048 after retry RPC@823070047), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@83835a5 -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910) -> (continuation of Deferred@2110108048 after retry RPC@823070047))) after lookup
16:16:31.001 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:31.002 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:31.002 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:31.002 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:31.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:31.039 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225),bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)[L],f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:31.039 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=192), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [16ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [16ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [32ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [34ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [37ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [37ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [51ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [53ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [55ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [55ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [151ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [153ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [155ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [155ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [191ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@422739586(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@32ccee3d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@18530678 after retry RPC@1219903999), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@83835a5 -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@18530678 after retry RPC@1219903999))) after lookup
16:16:31.040 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] connecting to peer
16:16:31.041 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:31.041 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] cleaning up while in state CONNECTING due to: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:31.041 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649) for tablet 8c1c2457ffe142cea0d61a82bffe854d: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649
16:16:31.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:31.966451 25389 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:16:31.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:31.966871 25389 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:31.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:31.969236 25389 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:31.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:31.992182 25038 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:16:32.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.002285 25326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38" is_pre_election: true
16:16:32.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.003369 25326 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 9.
16:16:32.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.005617 25036 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:16:32.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.006902 25389 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 10 FOLLOWER]: Leader pre-election won for term 11
16:16:32.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.007289 25389 raft_consensus.cc:482] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 10 FOLLOWER]: Starting leader election (detected failure of leader bdcfad5657c742588c55c4e90e2ac7fa)
16:16:32.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.007633 25389 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 10 FOLLOWER]: Advancing to term 11
16:16:32.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.015040 25389 raft_consensus.cc:504] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:32.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.019203 25326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1c2457ffe142cea0d61a82bffe854d" candidate_uuid: "3de21ca168134ac9a117e9e6cfc3a740" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "f679e1cd79344040853de5b0fe6e1b38"
16:16:32.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.020053 25326 raft_consensus.cc:3037] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 9 FOLLOWER]: Advancing to term 11
16:16:32.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:32.023519 25038 leader_election.cc:334] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111)
16:16:32.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.024704 25389 leader_election.cc:288] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 11 election: Requested vote from peers bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649), f679e1cd79344040853de5b0fe6e1b38 (127.22.31.66:39515)
16:16:32.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.032286 25326 raft_consensus.cc:2454] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de21ca168134ac9a117e9e6cfc3a740 in term 11.
16:16:32.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.033718 25036 leader_election.cc:302] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3de21ca168134ac9a117e9e6cfc3a740, f679e1cd79344040853de5b0fe6e1b38; no voters: bdcfad5657c742588c55c4e90e2ac7fa
16:16:32.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.035017 25389 raft_consensus.cc:2781] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 11 FOLLOWER]: Leader election won for term 11
16:16:32.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.036523 25389 raft_consensus.cc:686] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [term 11 LEADER]: Becoming Leader. State: Replica: 3de21ca168134ac9a117e9e6cfc3a740, State: Running, Role: LEADER
16:16:32.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.037580 25389 consensus_queue.cc:237] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:32.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.048089 22880 catalog_manager.cc:5495] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 reported cstate change: term changed from 10 to 11, leader changed from bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67) to 3de21ca168134ac9a117e9e6cfc3a740 (127.22.31.65). New cstate: current_term: 11 leader_uuid: "3de21ca168134ac9a117e9e6cfc3a740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } health_report { overall_health: HEALTHY } } }
16:16:32.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.523274 25326 raft_consensus.cc:1261] T 8c1c2457ffe142cea0d61a82bffe854d P f679e1cd79344040853de5b0fe6e1b38 [term 11 FOLLOWER]: Refusing update from remote peer 3de21ca168134ac9a117e9e6cfc3a740: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
16:16:32.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.525444 25389 consensus_queue.cc:1035] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.001s
16:16:32.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:32.567552 25038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111) [suppressed 2 similar messages]
16:16:32.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:32.572739 25038 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 -> Peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Couldn't send request to peer bdcfad5657c742588c55c4e90e2ac7fa. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:16:32.739 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 8c1c2457ffe142cea0d61a82bffe854d for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820' with partition [<start>, <end>)
16:16:32.740 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=8c1c2457ffe142cea0d61a82bffe854d@[3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)[L],bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649),f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)]}]
16:16:32.741 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=8c1c2457ffe142cea0d61a82bffe854d, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1893), Traces: [0ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [2ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [2ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [14ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [16ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [16ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [32ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [34ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [37ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [37ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [51ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [53ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [55ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [55ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [151ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [153ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [155ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [155ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [191ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, [193ms] sending RPC to server bdcfad5657c742588c55c4e90e2ac7fa, [194ms] received response from server bdcfad5657c742588c55c4e90e2ac7fa: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [194ms] delaying RPC due to: Network error: Failed to connect to peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649): Connection refused: /127.22.31.67:33649, [1887ms] refreshing cache from master, [1888ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.31.124:46519, [1892ms] Sub RPC GetTableLocations: received response from server master-127.22.31.124:46519: OK, deferred=Deferred@422739586(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$82/1697200096@32ccee3d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@18530678 after retry RPC@1219903999) -> (continuation of Deferred@1968413044 after retry RPC@1636604339), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@83835a5 -> wakeup thread Time-limited test -> (continuation of Deferred@274454369 after retry RPC@953010076) -> (continuation of Deferred@1531511044 after retry RPC@233757910) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@18530678 after retry RPC@1219903999) -> (continuation of Deferred@1968413044 after retry RPC@1636604339))) after lookup
16:16:32.742 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] connecting to peer
16:16:32.744 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] Successfully connected to peer
16:16:32.769 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x2bd4c77a, L:/127.0.0.1:60224 - R:/127.22.31.65:36225] using TOKEN/null
16:16:32.825 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.31.67:33649
16:16:32.826 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:32.828621 22653 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:32.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:16:32.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:32.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:32.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:32.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:16:33.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.233209 25425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:16:33.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.233871 25425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:16:33.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.234519 25425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:16:33.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.281738 25425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:16:33.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.282819 25425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.31.67
16:16:33.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:33.323478 25425 tablet_server_runner.cc:78] Tablet server non-default flags:
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.31.84:46853
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.31.67:33649
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.31.67
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36945
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.31.126:42753,127.22.31.125:39741,127.22.31.124:46519
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.31.67
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/logs
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:16:33.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:16:33.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:16:33.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 15:14:16 UTC on ccycloud.quasar-uwbvwl.root.comops.site
16:16:33.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580677
16:16:33.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:16:33.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:33.325017 25425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:16:33.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:33.327162 25425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:16:33.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.358891 25435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:34.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:34.390841 22917 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5bc0 after lost signal to thread 22835
16:16:34.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:34.392035 22917 debug-util.cc:398] Leaking SignalData structure 0x7b08000b8300 after lost signal to thread 22920
16:16:34.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.360723 25434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:34.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:33.362900 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:16:34.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:34.876389 25437 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1512 milliseconds
16:16:34.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.876549 25425 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:16:34.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.878189 25425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:16:34.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.882284 25425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:16:34.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.883790 25425 hybrid_clock.cc:648] HybridClock initialized: now 1786464994883740 us; error 86 us; skew 500 ppm
16:16:34.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.884938 25425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:16:34.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.902618 25425 webserver.cc:460] Webserver started at http://127.22.31.67:36945/ using document root <none> and password file <none>
16:16:34.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.903838 25425 fs_manager.cc:359] Metadata directory not provided
16:16:34.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.904116 25425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:16:34.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.916123 25425 fs_manager.cc:709] Time spent opening directory manager: real 0.007s	user 0.001s	sys 0.007s
16:16:34.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.922894 25453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
16:16:34.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.924851 25425 fs_manager.cc:725] Time spent opening block manager: real 0.006s	user 0.000s	sys 0.004s
16:16:34.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.925381 25425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data,/tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:34.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bdcfad5657c742588c55c4e90e2ac7fa"
16:16:34.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 16:15:46 on dist-test-slave-9jjr"
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.929644 25425 fs_report.cc:383] FS layout report
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/wal
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/data
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:16:34.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:16:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:34.930277 25038 consensus_peers.cc:487] T 8c1c2457ffe142cea0d61a82bffe854d P 3de21ca168134ac9a117e9e6cfc3a740 -> Peer bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649): Couldn't send request to peer bdcfad5657c742588c55c4e90e2ac7fa. Status: Network error: Client connection negotiation failed: client connection to 127.22.31.67:33649: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
16:16:34.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.988590 25425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
16:16:34.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.991034 25425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:16:34.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.991690 25425 kserver.cc:163] Server-wide thread pool size limit: 3276
16:16:34.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:34.995760 25425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:16:35.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.005754 25461 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
16:16:35.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.018153 25425 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
16:16:35.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.018936 25425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s	user 0.003s	sys 0.001s
16:16:35.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.019611 25425 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
16:16:35.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.029788 25425 ts_tablet_manager.cc:610] Registered 1 tablets
16:16:35.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.029904 25461 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap starting.
16:16:35.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.030634 25425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s	user 0.010s	sys 0.000s
16:16:35.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.130040 25461 log.cc:824] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Log is configured to *not* fsync() on all Append() calls
16:16:35.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.298012 25461 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:16:35.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.299412 25461 tablet_bootstrap.cc:492] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Bootstrap complete.
16:16:35.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.301614 25461 ts_tablet_manager.cc:1397] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent bootstrapping tablet: real 0.272s	user 0.208s	sys 0.056s
16:16:35.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.317233 25425 rpc_server.cc:307] RPC server started. Bound to: 127.22.31.67:33649
16:16:35.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.317499 25578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.31.67:33649 every 8 connection(s)
16:16:35.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.320811 25425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQnSfSC/test-tmp/mini-kudu-cluster1688455692877736157/ts-2/data/info.pb
16:16:35.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.326260 22653 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu as pid 25425
16:16:35.327 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
16:16:35.358 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:35.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.339161 25461 raft_consensus.cc:348] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:35.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.365022 25461 raft_consensus.cc:729] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcfad5657c742588c55c4e90e2ac7fa, State: Initialized, Role: FOLLOWER
16:16:35.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.372378 25461 consensus_queue.cc:260] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcfad5657c742588c55c4e90e2ac7fa" member_type: VOTER last_known_addr { host: "127.22.31.67" port: 33649 } } peers { permanent_uuid: "f679e1cd79344040853de5b0fe6e1b38" member_type: VOTER last_known_addr { host: "127.22.31.66" port: 39515 } } peers { permanent_uuid: "3de21ca168134ac9a117e9e6cfc3a740" member_type: VOTER last_known_addr { host: "127.22.31.65" port: 36225 } }
16:16:35.383 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786464946820, tablet=8c1c2457ffe142cea0d61a82bffe854d, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
16:16:35.402 [DEBUG - main] (Connection.java:575) [peer bdcfad5657c742588c55c4e90e2ac7fa(127.22.31.67:33649)] disconnecting while in state TERMINATED
16:16:35.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.404026 25579 heartbeater.cc:347] Connected to a master server at 127.22.31.124:46519
16:16:35.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.404696 25579 heartbeater.cc:464] Registering TS with master...
16:16:35.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.406224 25579 heartbeater.cc:511] Master 127.22.31.124:46519 requested a full tablet report, sending...
16:16:35.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.411410 22880 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:35.415 [DEBUG - main] (Connection.java:575) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] disconnecting while in state READY
16:16:35.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.420720 22880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.31.67:53275
16:16:35.422 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] handling channelInactive
16:16:35.423 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 3de21ca168134ac9a117e9e6cfc3a740(127.22.31.65:36225)] cleaning up while in state READY due to: connection closed
16:16:35.423 [DEBUG - main] (Connection.java:575) [peer master-127.22.31.124:46519(127.22.31.124:46519)] disconnecting while in state READY
16:16:35.425 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.22.31.124:46519(127.22.31.124:46519)] handling channelInactive
16:16:35.425 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.22.31.124:46519(127.22.31.124:46519)] cleaning up while in state READY due to: connection closed
16:16:35.427 [DEBUG - main] (Connection.java:575) [peer master-127.22.31.126:42753(127.22.31.126:42753)] disconnecting while in state READY
16:16:35.428 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.22.31.126:42753(127.22.31.126:42753)] handling channelInactive
16:16:35.429 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.22.31.126:42753(127.22.31.126:42753)] cleaning up while in state READY due to: connection closed
16:16:35.429 [DEBUG - main] (Connection.java:575) [peer f679e1cd79344040853de5b0fe6e1b38(127.22.31.66:39515)] disconnecting while in state TERMINATED
16:16:35.430 [DEBUG - main] (Connection.java:575) [peer master-127.22.31.125:39741(127.22.31.125:39741)] disconnecting while in state READY
16:16:35.432 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
16:16:35.434 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.22.31.125:39741(127.22.31.125:39741)] handling channelInactive
16:16:35.434 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.22.31.125:39741(127.22.31.125:39741)] cleaning up while in state READY due to: connection closed
16:16:35.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.438395 25580 heartbeater.cc:347] Connected to a master server at 127.22.31.125:39741
16:16:35.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.438987 25580 heartbeater.cc:464] Registering TS with master...
16:16:35.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.440073 25580 heartbeater.cc:511] Master 127.22.31.125:39741 requested a full tablet report, sending...
16:16:35.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.443437 25584 heartbeater.cc:347] Connected to a master server at 127.22.31.126:42753
16:16:35.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.443948 25584 heartbeater.cc:464] Registering TS with master...
16:16:35.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.444937 25584 heartbeater.cc:511] Master 127.22.31.126:42753 requested a full tablet report, sending...
16:16:35.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.444936 22790 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:35.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.449595 22701 ts_manager.cc:194] Re-registered known tserver with Master: bdcfad5657c742588c55c4e90e2ac7fa (127.22.31.67:33649)
16:16:35.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.451109 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 25011
16:16:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.456881 25579 heartbeater.cc:503] Master 127.22.31.124:46519 was elected leader, sending a full tablet report...
16:16:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.458310 25461 ts_tablet_manager.cc:1428] T 8c1c2457ffe142cea0d61a82bffe854d P bdcfad5657c742588c55c4e90e2ac7fa: Time spent starting tablet: real 0.156s	user 0.023s	sys 0.019s
16:16:35.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:35.510996 25593 negotiation.cc:337] Failed RPC negotiation. Trace:
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.410967 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.31.65:46375 (local address 127.22.31.67:33649)
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.427440 (+ 16473us) server_negotiation.cc:206] Beginning negotiation
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.427476 (+    36us) server_negotiation.cc:399] Waiting for connection header
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.427537 (+    61us) server_negotiation.cc:407] Connection header received
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.427731 (+   194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.427759 (+    28us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.428127 (+   368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.434973 (+  6846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.435791 (+   818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.438433 (+  2642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.447564 (+  9131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.450064 (+  2500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.451749 (+  1685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.452190 (+   441us) server_negotiation.cc:1035] Waiting for connection context
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 16:16:35.507776 (+ 55586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.31.65:46375: BlockingRecv error: recv got EOF from 127.22.31.65:46375 (error 108)
16:16:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15826,"thread_start_us":15459,"threads_started":1}
16:16:35.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.515235 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 25224
16:16:35.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.569869 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 25425
16:16:35.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.618360 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 22666
16:16:35.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.670929 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 22747
16:16:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:35.727485 22863 connection.cc:535] client connection to 127.22.31.125:39741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:16:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 16:16:35.728170 22863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:16:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 16:16:35.728317 22653 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQnSfSC/build/tsan/bin/kudu with pid 22830
16:16:37.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T16:16:37Z chronyd exiting

Time: 68.502

OK (1 test)

