JUnit version 4.13.2
.05:51:48.865 [INFO - main] (RandomUtils.java:49) Using random seed: 1783489908860
05:51:48.910 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
05:51:48.967 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
05:51:48.980 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskGJ_3KX/test-tmp
05:51:48.982 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
05:51:48.986 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskGJ_3KX/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
05:51:49.215 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

05:51:49.380 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
05:51:49.382 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

05:51:49.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.385331 11749 mini_oidc.cc:152] Starting JWKS server
05:51:49.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.388351 11749 webserver.cc:460] Webserver started at http://127.0.0.1:40025/ using document root <none> and password file <none>
05:51:49.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.388684 11749 mini_oidc.cc:189] Starting OIDC Discovery server
05:51:49.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.389411 11749 webserver.cc:460] Webserver started at http://127.0.0.1:34255/ using document root <none> and password file <none>
05:51:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T05:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
05:51:49.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T05:51:49Z Disabled control of system clock
05:51:49.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.420670 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:49.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:49.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/wal
05:51:49.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data
05:51:49.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:49.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:49.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:49.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:49.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:49.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:49.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/logs
05:51:49.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data/info.pb
05:51:49.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:49.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
05:51:49.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.126:37585
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.126
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:49.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:51:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:51:49.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:40025/jwks.json with env {}
05:51:49.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.734205 11760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:49.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.734495 11760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:49.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.734578 11760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:49.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.742132 11760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
05:51:49.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.742241 11760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:49.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.742285 11760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
05:51:49.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.742331 11760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
05:51:49.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.742367 11760 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
05:51:49.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.742421 11760 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:40025/jwks.json
05:51:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.756819 11760 master_runner.cc:386] Master server non-default flags:
05:51:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:49.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:49.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data
05:51:49.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/wal
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.126:37585
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:51:49.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:40025/jwks.json
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data/info.pb
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.126
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:49.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/logs
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:49.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:49.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:49.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:49.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:49.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.758560 11760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:49.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.760902 11760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:49.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.773707 11765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:49.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.773799 11766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:49.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.773806 11768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:49.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.774011 11760 server_base.cc:1034] running on GCE node
05:51:49.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.775364 11760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:49.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.776968 11760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:49.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.778235 11760 hybrid_clock.cc:648] HybridClock initialized: now 1783489909778152 us; error 43 us; skew 500 ppm
05:51:49.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.778636 11760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:49.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.789259 11760 webserver.cc:460] Webserver started at http://127.11.121.126:35221/ using document root <none> and password file <none>
05:51:49.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.790042 11760 fs_manager.cc:359] Metadata directory not provided
05:51:49.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.790202 11760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:49.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.790562 11760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:51:49.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.793357 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data/instance:
05:51:49.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3a64352b29874c3f8f14d5019b205f16"
05:51:49.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:49 on dist-test-slave-w6r9"
05:51:49.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.794302 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/wal/instance:
05:51:49.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3a64352b29874c3f8f14d5019b205f16"
05:51:49.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:49 on dist-test-slave-w6r9"
05:51:49.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.799671 11760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
05:51:49.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.803632 11774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:49.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.805054 11760 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:51:49.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.805263 11760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/wal
05:51:49.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3a64352b29874c3f8f14d5019b205f16"
05:51:49.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:49 on dist-test-slave-w6r9"
05:51:49.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.805536 11760 fs_report.cc:383] FS layout report
05:51:49.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/wal
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/wal
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data/data
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:49.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:49.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:49.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:49.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:49.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:49.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:49.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:49.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.827531 11760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.828529 11760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.828895 11760 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.848678 11760 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.126:37585
05:51:49.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.848696 11825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.126:37585 every 8 connection(s)
05:51:49.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.850414 11760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/data/info.pb
05:51:49.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.854588 11826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
05:51:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.858428 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 11760
05:51:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.858709 11749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-0/wal/instance
05:51:49.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.861511 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:49.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:49.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/wal
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/logs
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data/info.pb
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:49.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.125:38073
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.125
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:51:49.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:49.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:51:49.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:40025/jwks.json with env {}
05:51:49.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.862380 11826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
05:51:49.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.883358 11826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
05:51:49.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.884742 11778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.125:38073: connect: Connection refused (error 111)
05:51:49.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.886482 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.125:38073: Network error: Client connection negotiation failed: client connection to 127.11.121.125:38073: connect: Connection refused (error 111)
05:51:49.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:49.935829 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } attempt: 1
05:51:49.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:49.937914 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.125:38073: Network error: Client connection negotiation failed: client connection to 127.11.121.125:38073: connect: Connection refused (error 111)
05:51:50.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.006479 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } attempt: 2
05:51:50.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.009086 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.125:38073: Network error: Client connection negotiation failed: client connection to 127.11.121.125:38073: connect: Connection refused (error 111)
05:51:50.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.100375 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } attempt: 3
05:51:50.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.102753 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.125:38073: Network error: Client connection negotiation failed: client connection to 127.11.121.125:38073: connect: Connection refused (error 111)
05:51:50.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.225453 11828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:50.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.225749 11828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:50.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.225831 11828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:50.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.233273 11828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
05:51:50.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.233387 11828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:50.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.233428 11828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
05:51:50.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.233475 11828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
05:51:50.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.233511 11828 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
05:51:50.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.233559 11828 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:40025/jwks.json
05:51:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.242895 11828 master_runner.cc:386] Master server non-default flags:
05:51:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data
05:51:50.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/wal
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.125:38073
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:51:50.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:40025/jwks.json
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data/info.pb
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.125
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/logs
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:50.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:50.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:50.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
05:51:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:50.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.244359 11828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:50.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.246022 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } attempt: 4
05:51:50.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.246192 11828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:50.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.248395 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.125:38073: Network error: Client connection negotiation failed: client connection to 127.11.121.125:38073: connect: Connection refused (error 111)
05:51:50.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.259344 11836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:50.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.259320 11837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:50.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.259239 11839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:50.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.260483 11828 server_base.cc:1034] running on GCE node
05:51:50.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.261322 11828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:50.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.262717 11828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.264016 11828 hybrid_clock.cc:648] HybridClock initialized: now 1783489910263933 us; error 59 us; skew 500 ppm
05:51:50.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.264434 11828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:50.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.266986 11828 webserver.cc:460] Webserver started at http://127.11.121.125:38149/ using document root <none> and password file <none>
05:51:50.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.267797 11828 fs_manager.cc:359] Metadata directory not provided
05:51:50.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.267969 11828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:50.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.268417 11828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:51:50.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.271876 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data/instance:
05:51:50.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3fd9807d6070412c9daf232f3b9859d1"
05:51:50.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:50 on dist-test-slave-w6r9"
05:51:50.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.272969 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/wal/instance:
05:51:50.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3fd9807d6070412c9daf232f3b9859d1"
05:51:50.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:50 on dist-test-slave-w6r9"
05:51:50.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.278756 11828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
05:51:50.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.282963 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:50.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.284507 11828 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.284729 11828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/wal
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3fd9807d6070412c9daf232f3b9859d1"
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:50 on dist-test-slave-w6r9"
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.284987 11828 fs_report.cc:383] FS layout report
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/wal
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/wal
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data/data
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:50.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:50.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:50.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:50.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:50.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.311480 11828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:50.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.312475 11828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:50.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.312873 11828 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:50.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.331076 11828 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.125:38073
05:51:50.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.331221 11896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.125:38073 every 8 connection(s)
05:51:50.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.332579 11828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/data/info.pb
05:51:50.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.336609 11897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
05:51:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.340101 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 11828
05:51:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.340289 11749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-1/wal/instance
05:51:50.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.341691 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:50.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/wal
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/logs
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data/info.pb
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.124:33337
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.124
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:51:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:51:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:40025/jwks.json with env {}
05:51:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.345405 11897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
05:51:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.360522 11897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
05:51:50.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.371984 11897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } has no permanent_uuid. Determining permanent_uuid...
05:51:50.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.373142 11848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.374816 11897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.424153 11897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 1
05:51:50.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.426501 11897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.494800 11897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 2
05:51:50.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.496970 11897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.547659 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } attempt: 5
05:51:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.555023 11826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } has no permanent_uuid. Determining permanent_uuid...
05:51:50.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.557288 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.588261 11897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 3
05:51:50.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.590806 11897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.608597 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 1
05:51:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.610742 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.679039 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 2
05:51:50.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.681578 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.711261 11899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:50.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.711576 11899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:50.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.711670 11899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:50.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.719550 11899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
05:51:50.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.719653 11899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:50.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.719696 11899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
05:51:50.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.719744 11899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
05:51:50.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.719781 11899 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
05:51:50.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.719830 11899 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:40025/jwks.json
05:51:50.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.729589 11899 master_runner.cc:386] Master server non-default flags:
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/wal
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.124:33337
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:40025/jwks.json
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data/info.pb
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.124
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:50.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/logs
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:50.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
05:51:50.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:50.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:50.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:50.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:50.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:50.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:50.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.730839 11899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:50.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.733237 11899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:50.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.734164 11897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 4
05:51:50.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.736371 11897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.746444 11907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:50.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.746517 11908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:50.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.746543 11910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:50.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.747710 11899 server_base.cc:1034] running on GCE node
05:51:50.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.748587 11899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:50.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.750002 11899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:50.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.751248 11899 hybrid_clock.cc:648] HybridClock initialized: now 1783489910751161 us; error 52 us; skew 500 ppm
05:51:50.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.751607 11899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:50.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.754276 11899 webserver.cc:460] Webserver started at http://127.11.121.124:42757/ using document root <none> and password file <none>
05:51:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.754957 11899 fs_manager.cc:359] Metadata directory not provided
05:51:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.755064 11899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.755322 11899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:51:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.758154 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data/instance:
05:51:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "84db27231add4d86ae0c9105b822b26f"
05:51:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:50 on dist-test-slave-w6r9"
05:51:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.758970 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/wal/instance:
05:51:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "84db27231add4d86ae0c9105b822b26f"
05:51:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:50 on dist-test-slave-w6r9"
05:51:50.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.764827 11899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
05:51:50.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.768509 11916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.769997 11899 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.770138 11899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/wal
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "84db27231add4d86ae0c9105b822b26f"
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:50 on dist-test-slave-w6r9"
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.770308 11899 fs_report.cc:383] FS layout report
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/wal
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/wal
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data/data
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:50.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:50.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:50.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:50.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.783564 11899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:50.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.784420 11899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:50.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.784703 11899 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:50.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.787889 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 3
05:51:50.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.790086 11826 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.121.124:33337: Network error: Client connection negotiation failed: client connection to 127.11.121.124:33337: connect: Connection refused (error 111)
05:51:50.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.803799 11899 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.124:33337
05:51:50.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.803833 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.124:33337 every 8 connection(s)
05:51:50.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.805271 11899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/data/info.pb
05:51:50.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.809237 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 11899
05:51:50.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.809399 11749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/master-2/wal/instance
05:51:50.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.809440 11968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
05:51:50.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.816730 11968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
05:51:50.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.831997 11968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
05:51:50.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.839679 11968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } has no permanent_uuid. Determining permanent_uuid...
05:51:50.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.852021 11968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f: Bootstrap starting.
05:51:50.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.856062 11968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f: Neither blocks nor log segments found. Creating new log.
05:51:50.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.857419 11968 log.cc:824] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f: Log is configured to *not* fsync() on all Append() calls
05:51:50.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.860401 11968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f: No bootstrap required, opened a new log
05:51:50.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.867746 11968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:50.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.868121 11968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:51:50.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.868203 11968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84db27231add4d86ae0c9105b822b26f, State: Initialized, Role: FOLLOWER
05:51:50.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.868777 11968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:50.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.869671 11975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:50.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.870093 11975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: This master's current role is: FOLLOWER
05:51:50.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.870558 11968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: configured and running, proceeding with master startup.
05:51:50.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.876982 11899 master_runner.cc:186] Error getting master registration for 127.11.121.126:37585: OK, instance_id { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" instance_seqno: 1783489909842948 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
05:51:50.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.884207 11899 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
05:51:50.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.887874 11986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
05:51:50.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:50.888274 11986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
05:51:50.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.967332 11826 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 4
05:51:50.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.980763 11826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: Bootstrap starting.
05:51:50.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.985636 11826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: Neither blocks nor log segments found. Creating new log.
05:51:50.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.987037 11826 log.cc:824] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: Log is configured to *not* fsync() on all Append() calls
05:51:50.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.990357 11826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: No bootstrap required, opened a new log
05:51:50.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.998896 11826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:50.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.999274 11826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:51:50.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:50.999396 11826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a64352b29874c3f8f14d5019b205f16, State: Initialized, Role: FOLLOWER
05:51:51.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.000217 11826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:51.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.001238 11989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.001891 11989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [sys.catalog]: This master's current role is: FOLLOWER
05:51:51.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.002805 11826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [sys.catalog]: configured and running, proceeding with master startup.
05:51:51.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.014816 12000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
05:51:51.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.015329 12000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
05:51:51.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.012873 11760 master_runner.cc:186] Error getting master registration for 127.11.121.125:38073: OK, instance_id { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" instance_seqno: 1783489910326044 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
05:51:51.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.026952 11760 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
05:51:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.035610 11897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } attempt: 5
05:51:51.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.036656 11922 tablet.cc:2369] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:51:51.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.051862 11897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1: Bootstrap starting.
05:51:51.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.058234 11897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1: Neither blocks nor log segments found. Creating new log.
05:51:51.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.061319 11897 log.cc:824] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1: Log is configured to *not* fsync() on all Append() calls
05:51:51.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.069802 11897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1: No bootstrap required, opened a new log
05:51:51.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.082041 11780 tablet.cc:2369] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:51:51.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.082419 11897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:51.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.082836 11897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:51:51.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.082942 11897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd9807d6070412c9daf232f3b9859d1, State: Initialized, Role: FOLLOWER
05:51:51.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.083709 11897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.089341 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.090271 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [sys.catalog]: This master's current role is: FOLLOWER
05:51:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.090281 11975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:51:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.090531 11975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:51.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.090950 11897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [sys.catalog]: configured and running, proceeding with master startup.
05:51:51.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.092099 11975 leader_election.cc:288] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a64352b29874c3f8f14d5019b205f16 (127.11.121.126:37585), 3fd9807d6070412c9daf232f3b9859d1 (127.11.121.125:38073)
05:51:51.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.093442 11801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84db27231add4d86ae0c9105b822b26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a64352b29874c3f8f14d5019b205f16" is_pre_election: true
05:51:51.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.094084 11801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84db27231add4d86ae0c9105b822b26f in term 0.
05:51:51.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.094769 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84db27231add4d86ae0c9105b822b26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd9807d6070412c9daf232f3b9859d1" is_pre_election: true
05:51:51.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.095289 11872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84db27231add4d86ae0c9105b822b26f in term 0.
05:51:51.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.095902 11920 leader_election.cc:302] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd9807d6070412c9daf232f3b9859d1, 84db27231add4d86ae0c9105b822b26f; no voters: 
05:51:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.097249 11975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Leader pre-election won for term 1
05:51:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.097401 11975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
05:51:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.097489 11975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 0 FOLLOWER]: Advancing to term 1
05:51:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.100561 11975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.102394 11801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84db27231add4d86ae0c9105b822b26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a64352b29874c3f8f14d5019b205f16"
05:51:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.102623 11801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [term 0 FOLLOWER]: Advancing to term 1
05:51:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.105897 11801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84db27231add4d86ae0c9105b822b26f in term 1.
05:51:51.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.106338 11919 leader_election.cc:302] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a64352b29874c3f8f14d5019b205f16, 84db27231add4d86ae0c9105b822b26f; no voters: 
05:51:51.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.110878 12007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 1 FOLLOWER]: Leader election won for term 1
05:51:51.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.111387 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84db27231add4d86ae0c9105b822b26f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd9807d6070412c9daf232f3b9859d1"
05:51:51.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.111627 11872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [term 0 FOLLOWER]: Advancing to term 1
05:51:51.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.114149 11975 leader_election.cc:288] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [CANDIDATE]: Term 1 election: Requested vote from peers 3a64352b29874c3f8f14d5019b205f16 (127.11.121.126:37585), 3fd9807d6070412c9daf232f3b9859d1 (127.11.121.125:38073)
05:51:51.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.119175 11872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84db27231add4d86ae0c9105b822b26f in term 1.
05:51:51.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.120872 12007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [term 1 LEADER]: Becoming Leader. State: Replica: 84db27231add4d86ae0c9105b822b26f, State: Running, Role: LEADER
05:51:51.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.122056 12007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } }
05:51:51.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.140540 11975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84db27231add4d86ae0c9105b822b26f. Latest consensus state: current_term: 1 leader_uuid: "84db27231add4d86ae0c9105b822b26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.141023 11975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: This master's current role is: LEADER
05:51:51.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.155311 12012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
05:51:51.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.158847 12012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
05:51:51.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.182528 11872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [term 1 FOLLOWER]: Refusing update from remote peer 84db27231add4d86ae0c9105b822b26f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:51:51.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.184249 11975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
05:51:51.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.187822 12019 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
05:51:51.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.196830 11801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [term 1 FOLLOWER]: Refusing update from remote peer 84db27231add4d86ae0c9105b822b26f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:51:51.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.197757 11975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
05:51:51.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.206121 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84db27231add4d86ae0c9105b822b26f. Latest consensus state: current_term: 1 leader_uuid: "84db27231add4d86ae0c9105b822b26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.206404 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [sys.catalog]: This master's current role is: FOLLOWER
05:51:51.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.212379 12017 mvcc.cc:205] Tried to move back new op lower bound from 7305174676188958720 to 7305174675978297344. Current Snapshot: MvccSnapshot[applied={T|T < 7305174676188958720}]
05:51:51.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.232262 12007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84db27231add4d86ae0c9105b822b26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.232618 12007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: This master's current role is: LEADER
05:51:51.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.236963 11989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84db27231add4d86ae0c9105b822b26f. Latest consensus state: current_term: 1 leader_uuid: "84db27231add4d86ae0c9105b822b26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.237356 11989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [sys.catalog]: This master's current role is: FOLLOWER
05:51:51.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.240459 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84db27231add4d86ae0c9105b822b26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.240751 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1 [sys.catalog]: This master's current role is: FOLLOWER
05:51:51.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.242607 12012 catalog_manager.cc:1347] Generated new cluster ID: 79415f99479a4bf080a9340b320dfbe0
05:51:51.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.242717 12012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
05:51:51.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.251231 11975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84db27231add4d86ae0c9105b822b26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.251550 11975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f [sys.catalog]: This master's current role is: LEADER
05:51:51.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.256747 12022 mvcc.cc:205] Tried to move back new op lower bound from 7305174676188958720 to 7305174675978297344. Current Snapshot: MvccSnapshot[applied={T|T < 7305174676188958720}]
05:51:51.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.264499 11989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84db27231add4d86ae0c9105b822b26f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a64352b29874c3f8f14d5019b205f16" member_type: VOTER last_known_addr { host: "127.11.121.126" port: 37585 } } peers { permanent_uuid: "3fd9807d6070412c9daf232f3b9859d1" member_type: VOTER last_known_addr { host: "127.11.121.125" port: 38073 } } peers { permanent_uuid: "84db27231add4d86ae0c9105b822b26f" member_type: VOTER last_known_addr { host: "127.11.121.124" port: 33337 } } }
05:51:51.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.264787 11989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16 [sys.catalog]: This master's current role is: FOLLOWER
05:51:51.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.270424 12012 catalog_manager.cc:1370] Generated new certificate authority record
05:51:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.272262 12012 catalog_manager.cc:1504] Loading token signing keys...
05:51:51.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.299453 12012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f: Generated new TSK 0
05:51:51.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.301048 12012 catalog_manager.cc:1514] Initializing in-progress tserver states...
05:51:51.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.310712 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:51.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:51.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:51:51.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:51:51.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:51.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:0
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:51.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:51.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:51.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:51:51.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.669379 12027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:51.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.669718 12027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:51.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.669834 12027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:51.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.678179 12027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:51.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.678422 12027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.65
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.687815 12027 tablet_server_runner.cc:78] Tablet server non-default flags:
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:0
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:51.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:51.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:51.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.689628 12027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:51.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.691689 12027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:51.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.706730 12032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:51.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.706700 12033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:51.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:51.706919 12035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:51.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.708271 12027 server_base.cc:1034] running on GCE node
05:51:51.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.708858 12027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:51.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.709820 12027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:51.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.711043 12027 hybrid_clock.cc:648] HybridClock initialized: now 1783489911710955 us; error 137 us; skew 500 ppm
05:51:51.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.711403 12027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:51.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.713696 12027 webserver.cc:460] Webserver started at http://127.11.121.65:34993/ using document root <none> and password file <none>
05:51:51.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.714323 12027 fs_manager.cc:359] Metadata directory not provided
05:51:51.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.714417 12027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:51.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.714653 12027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:51:51.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.717275 12027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/instance:
05:51:51.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:51:51.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:51 on dist-test-slave-w6r9"
05:51:51.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.717980 12027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal/instance:
05:51:51.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:51:51.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:51 on dist-test-slave-w6r9"
05:51:51.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.722997 12027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.005s
05:51:51.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.726354 12041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.728066 12027 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.728207 12027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:51 on dist-test-slave-w6r9"
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.728370 12027 fs_report.cc:383] FS layout report
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/data
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:51.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:51.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:51.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.762619 12027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.763445 12027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.763700 12027 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:51.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.764724 12027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:51:51.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.766229 12027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
05:51:51.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.766300 12027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
05:51:51.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.766407 12027 ts_tablet_manager.cc:610] Registered 0 tablets
05:51:51.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.766445 12027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
05:51:51.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.811228 12027 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.65:46477
05:51:51.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.811213 12153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.65:46477 every 8 connection(s)
05:51:51.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.813298 12027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:51:51.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.823920 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 12027
05:51:51.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.824174 11749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal/instance
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.831033 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:51:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:0
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:51:51.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.858072 12154 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:51:51.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.858374 12154 heartbeater.cc:464] Registering TS with master...
05:51:51.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.859406 12154 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:51:51.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.861155 12155 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:51:51.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.861263 12155 heartbeater.cc:464] Registering TS with master...
05:51:51.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.861706 12155 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:51:51.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.863863 11933 ts_manager.cc:194] Registered new tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:51.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.863965 11791 ts_manager.cc:194] Registered new tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:51.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.865907 11933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.65:51639
05:51:51.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.870069 12156 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:51:51.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.870178 12156 heartbeater.cc:464] Registering TS with master...
05:51:51.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.870548 12156 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:51:51.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:51.872696 11862 ts_manager.cc:194] Registered new tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:52.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.017962 12000 catalog_manager.cc:1259] Loaded cluster ID: 79415f99479a4bf080a9340b320dfbe0
05:51:52.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.018105 12000 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: loading cluster ID for follower catalog manager: success
05:51:52.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.021577 12000 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: acquiring CA information for follower catalog manager: success
05:51:52.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.023932 12000 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a64352b29874c3f8f14d5019b205f16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
05:51:52.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.184820 12162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:52.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.185158 12162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:52.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.185281 12162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:52.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.193831 12019 catalog_manager.cc:1259] Loaded cluster ID: 79415f99479a4bf080a9340b320dfbe0
05:51:52.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.194339 12019 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1: loading cluster ID for follower catalog manager: success
05:51:52.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.194530 12162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:52.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.194844 12162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.66
05:51:52.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.197854 12019 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1: acquiring CA information for follower catalog manager: success
05:51:52.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.200148 12019 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fd9807d6070412c9daf232f3b9859d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
05:51:52.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.209656 12162 tablet_server_runner.cc:78] Tablet server non-default flags:
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:0
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:52.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:52.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.211314 12162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:52.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.213732 12162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:52.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.232344 12171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:52.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.232595 12173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:52.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.233068 12170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.233902 12162 server_base.cc:1034] running on GCE node
05:51:52.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.234660 12162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:52.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.236616 12162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:52.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.237874 12162 hybrid_clock.cc:648] HybridClock initialized: now 1783489912237739 us; error 105 us; skew 500 ppm
05:51:52.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.238271 12162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:52.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.244182 12162 webserver.cc:460] Webserver started at http://127.11.121.66:43205/ using document root <none> and password file <none>
05:51:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.245165 12162 fs_manager.cc:359] Metadata directory not provided
05:51:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.245321 12162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:52.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.245599 12162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:51:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.248451 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/instance:
05:51:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abf53778adcf497787bbf7bac00509e2"
05:51:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.249414 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal/instance:
05:51:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abf53778adcf497787bbf7bac00509e2"
05:51:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:52.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.255489 12162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.005s
05:51:52.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.259331 12179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.261006 12162 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.261251 12162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abf53778adcf497787bbf7bac00509e2"
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.261526 12162 fs_report.cc:383] FS layout report
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:52.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/data
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:52.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:52.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.278616 12162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:52.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.279651 12162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:52.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.280040 12162 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:52.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.281179 12162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:51:52.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.282954 12162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
05:51:52.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.283077 12162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
05:51:52.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.283217 12162 ts_tablet_manager.cc:610] Registered 0 tablets
05:51:52.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.283299 12162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
05:51:52.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.346956 12162 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.66:39283
05:51:52.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.346974 12291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.66:39283 every 8 connection(s)
05:51:52.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.348973 12162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:51:52.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.359308 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 12162
05:51:52.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.359488 11749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal/instance
05:51:52.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.368116 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:52.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:52.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:52.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:0
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:52.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:52.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:52.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:52.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:51:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.376823 12293 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:51:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.377200 12293 heartbeater.cc:464] Registering TS with master...
05:51:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.378347 12293 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:51:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.380448 12294 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:51:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.380584 12294 heartbeater.cc:464] Registering TS with master...
05:51:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.380530 11791 ts_manager.cc:194] Registered new tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:51:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.381063 12294 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:51:52.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.382100 11862 ts_manager.cc:194] Registered new tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:51:52.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.382913 12292 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:51:52.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.383062 12292 heartbeater.cc:464] Registering TS with master...
05:51:52.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.383556 12292 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:51:52.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.384584 11933 ts_manager.cc:194] Registered new tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:51:52.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.385828 11933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.66:43417
05:51:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.716491 12299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:52.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.716823 12299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:52.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.716938 12299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.724382 12299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.724633 12299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.67
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.734165 12299 tablet_server_runner.cc:78] Tablet server non-default flags:
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:0
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:52.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:51:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:52.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.736003 12299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.737831 12299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.751930 12305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.751999 12304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:52.752013 12307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:52.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.753850 12299 server_base.cc:1034] running on GCE node
05:51:52.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.754581 12299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:52.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.755620 12299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:52.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.756862 12299 hybrid_clock.cc:648] HybridClock initialized: now 1783489912756797 us; error 34 us; skew 500 ppm
05:51:52.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.757215 12299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:52.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.759557 12299 webserver.cc:460] Webserver started at http://127.11.121.67:40053/ using document root <none> and password file <none>
05:51:52.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.760321 12299 fs_manager.cc:359] Metadata directory not provided
05:51:52.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.760469 12299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:52.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.760905 12299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
05:51:52.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.763559 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/instance:
05:51:52.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7c5d736667634e189b1c959e88b14aba"
05:51:52.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:52.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.764375 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal/instance:
05:51:52.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7c5d736667634e189b1c959e88b14aba"
05:51:52.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:52.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.769663 12299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.002s
05:51:52.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.773170 12313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:52.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.774546 12299 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
05:51:52.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.774739 12299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7c5d736667634e189b1c959e88b14aba"
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.774963 12299 fs_report.cc:383] FS layout report
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/data
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:52.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:52.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:52.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:52.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:52.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.790516 12299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:52.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.791378 12299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:52.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.791771 12299 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:52.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.792934 12299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:51:52.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.794528 12299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
05:51:52.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.794634 12299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
05:51:52.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.794771 12299 ts_tablet_manager.cc:610] Registered 0 tablets
05:51:52.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.794835 12299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
05:51:52.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.834774 12299 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.67:38133
05:51:52.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.834816 12425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.67:38133 every 8 connection(s)
05:51:52.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.836195 12299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:51:52.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.836735 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 12299
05:51:52.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.836942 11749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal/instance
05:51:52.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.857468 12427 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:51:52.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.857818 12428 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:51:52.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.857905 12426 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:51:52.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.857962 12428 heartbeater.cc:464] Registering TS with master...
05:51:52.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.858027 12426 heartbeater.cc:464] Registering TS with master...
05:51:52.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.858991 12426 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:51:52.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.859022 12428 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:51:52.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.857848 12427 heartbeater.cc:464] Registering TS with master...
05:51:52.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.860426 12427 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:51:52.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.860905 11862 ts_manager.cc:194] Registered new tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:52.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.861827 11791 ts_manager.cc:194] Registered new tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:52.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.862939 11933 ts_manager.cc:194] Registered new tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:52.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.863970 11933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.67:38085
05:51:52.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.869127 12154 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:51:52.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:52.878135 11749 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
05:51:52.880 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
05:51:52.881 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

05:51:52.893 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.11.121.126"
      port: 37585
    }
    bound_http_address {
      host: "127.11.121.126"
      port: 35221
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.11.121.125"
      port: 38073
    }
    bound_http_address {
      host: "127.11.121.125"
      port: 38149
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.11.121.124"
      port: 33337
    }
    bound_http_address {
      host: "127.11.121.124"
      port: 42757
    }
  }
}

05:51:52.901 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

05:51:52.904 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.11.121.65"
      port: 46477
    }
    bound_http_address {
      host: "127.11.121.65"
      port: 34993
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.11.121.66"
      port: 39283
    }
    bound_http_address {
      host: "127.11.121.66"
      port: 43205
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.11.121.67"
      port: 38133
    }
    bound_http_address {
      host: "127.11.121.67"
      port: 40053
    }
  }
}

05:51:52.905 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
05:51:53.078 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
05:51:53.271 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.11.121.124:33337(127.11.121.124:33337)] connecting to peer
05:51:53.343 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.11.121.126:37585(127.11.121.126:37585)] connecting to peer
05:51:53.348 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.11.121.125:38073(127.11.121.125:38073)] connecting to peer
05:51:53.380 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.11.121.126:37585(127.11.121.126:37585)] Successfully connected to peer
05:51:53.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.389114 12292 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:51:53.391 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.11.121.124:33337(127.11.121.124:33337)] Successfully connected to peer
05:51:53.463 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.11.121.125:38073(127.11.121.125:38073)] Successfully connected to peer
05:51:53.487 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.11.121.124
05:51:53.487 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.11.121.126
05:51:53.589 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.11.121.125
05:51:53.703 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x0ca8af82, L:/127.0.0.1:58530 - R:/127.11.121.124:33337] using SASL/PLAIN
05:51:53.703 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x471bb10a, L:/127.0.0.1:32834 - R:/127.11.121.126:37585] using SASL/PLAIN
05:51:53.707 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x01cbddcf, L:/127.0.0.1:44714 - R:/127.11.121.125:38073] using SASL/PLAIN
05:51:53.745 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
05:51:53.746 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=527), Traces: [0ms] refreshing cache from master, [36ms] Sub RPC ConnectToMaster: sending RPC to server master-127.11.121.124:33337, [119ms] Sub RPC ConnectToMaster: sending RPC to server master-127.11.121.126:37585, [124ms] Sub RPC ConnectToMaster: sending RPC to server master-127.11.121.125:38073, [493ms] Sub RPC ConnectToMaster: received response from server master-127.11.121.126:37585: OK, [494ms] Sub RPC ConnectToMaster: received response from server master-127.11.121.124:33337: OK, deferred=null) after lookup
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.796265 11933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58530:
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204"
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
05:51:53.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
05:51:53.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
05:51:53.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
05:51:53.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
05:51:53.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
05:51:53.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:53.798553 11933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
05:51:53.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.866639 12426 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:51:53.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.890407 12361 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204 [id=66ff539e739145079935576329723008]), partition=RANGE (key) PARTITION UNBOUNDED
05:51:53.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.891987 12361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f. 1 dirs total, 0 dirs full, 0 dirs failed
05:51:53.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.894577 12089 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204 [id=66ff539e739145079935576329723008]), partition=RANGE (key) PARTITION UNBOUNDED
05:51:53.896 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 66ff539e739145079935576329723008
05:51:53.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.896166 12089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f. 1 dirs total, 0 dirs full, 0 dirs failed
05:51:53.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.910122 12227 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204 [id=66ff539e739145079935576329723008]), partition=RANGE (key) PARTITION UNBOUNDED
05:51:53.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.911847 12227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f. 1 dirs total, 0 dirs full, 0 dirs failed
05:51:53.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.914510 12457 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap starting.
05:51:53.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.919155 12457 tablet_bootstrap.cc:654] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Neither blocks nor log segments found. Creating new log.
05:51:53.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.920931 12458 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap starting.
05:51:53.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.921018 12457 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Log is configured to *not* fsync() on all Append() calls
05:51:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.925071 12458 tablet_bootstrap.cc:654] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Neither blocks nor log segments found. Creating new log.
05:51:53.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.926597 12458 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Log is configured to *not* fsync() on all Append() calls
05:51:53.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.930791 12457 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: No bootstrap required, opened a new log
05:51:53.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.931317 12457 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent bootstrapping tablet: real 0.017s	user 0.000s	sys 0.016s
05:51:53.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.931628 12458 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: No bootstrap required, opened a new log
05:51:53.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.932116 12458 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
05:51:53.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.942543 12457 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:53.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.944011 12461 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap starting.
05:51:53.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.945797 12458 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:53.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.946238 12458 raft_consensus.cc:374] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:51:53.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.946388 12457 raft_consensus.cc:374] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:51:53.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.946388 12458 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f80c10be94511bdabb93db3d6d160, State: Initialized, Role: FOLLOWER
05:51:53.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.946484 12457 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Initialized, Role: FOLLOWER
05:51:53.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.947146 12457 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:53.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.947146 12458 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:53.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.947714 12461 tablet_bootstrap.cc:654] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Neither blocks nor log segments found. Creating new log.
05:51:53.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.948945 12461 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Log is configured to *not* fsync() on all Append() calls
05:51:53.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.951339 12457 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.007s
05:51:53.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.951718 12461 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: No bootstrap required, opened a new log
05:51:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.952131 12461 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent bootstrapping tablet: real 0.008s	user 0.008s	sys 0.000s
05:51:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.958976 12458 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent starting tablet: real 0.027s	user 0.021s	sys 0.004s
05:51:53.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.962129 12461 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:53.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.962783 12461 raft_consensus.cc:374] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
05:51:53.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.963053 12461 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Initialized, Role: FOLLOWER
05:51:53.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.963713 12461 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:53.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.965761 12461 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent starting tablet: real 0.014s	user 0.012s	sys 0.000s
05:51:53.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.998108 12465 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:51:53.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:53.998546 12465 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:54.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.000764 12465 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:54.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.009460 12381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba" is_pre_election: true
05:51:54.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.009464 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160" is_pre_election: true
05:51:54.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.009985 12381 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 0.
05:51:54.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.010002 12109 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 0.
05:51:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.010813 12182 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2; no voters: 
05:51:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.011303 12465 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
05:51:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.011504 12465 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
05:51:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.011637 12465 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 0 FOLLOWER]: Advancing to term 1
05:51:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.013841 12465 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.014570 12465 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.015039 12381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba"
05:51:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.015152 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:51:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.015277 12381 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 0 FOLLOWER]: Advancing to term 1
05:51:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.015349 12109 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 0 FOLLOWER]: Advancing to term 1
05:51:54.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.018648 12109 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 1.
05:51:54.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.018648 12381 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 1.
05:51:54.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.019167 12183 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 
05:51:54.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.019506 12465 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Leader election won for term 1
05:51:54.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.020117 12465 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 LEADER]: Becoming Leader. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Running, Role: LEADER
05:51:54.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.020959 12465 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:54.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.026643 11932 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to abf53778adcf497787bbf7bac00509e2 (127.11.121.66). New cstate: current_term: 1 leader_uuid: "abf53778adcf497787bbf7bac00509e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:51:54.048 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 66ff539e739145079935576329723008
05:51:54.068 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:54.075 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299992, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:51:54.077 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [8ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=null) after lookup
05:51:54.088 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:51:54.090 [DEBUG - kudu-nio-1] (Connection.java:757) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] Successfully connected to peer
05:51:54.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:54.091857 12429 tablet.cc:2369] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:51:54.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:54.101761 12157 tablet.cc:2369] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:51:54.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:54.128378 12298 tablet.cc:2369] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
05:51:54.151 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xa9c1d3f6, L:/127.0.0.1:48862 - R:/127.11.121.66:39283] using TOKEN/null
05:51:54.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.162117 12381 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 1 FOLLOWER]: Refusing update from remote peer abf53778adcf497787bbf7bac00509e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:51:54.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.162117 12109 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 1 FOLLOWER]: Refusing update from remote peer abf53778adcf497787bbf7bac00509e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
05:51:54.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.163033 12468 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
05:51:54.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.163381 12465 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
05:51:54.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.179517 12470 mvcc.cc:205] Tried to move back new op lower bound from 7305174688396652544 to 7305174687835111424. Current Snapshot: MvccSnapshot[applied={T|T < 7305174688396652544}]
05:51:54.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.189720 12472 mvcc.cc:205] Tried to move back new op lower bound from 7305174688396652544 to 7305174687835111424. Current Snapshot: MvccSnapshot[applied={T|T < 7305174688396652544 or (T in {7305174688396652544})}]
05:51:54.265 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:51:54.267 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.66:39283
05:51:54.269 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:51:54.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.269811 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 12162
05:51:54.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.279564 12287 generic_service.cc:196] Checking for leaks (request via RPC)
05:51:54.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:54.377727 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 12162
05:51:54.400 [DEBUG - kudu-nio-1] (Connection.java:239) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] handling channelInactive
05:51:54.401 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:51:54.401 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state READY due to: connection closed
05:51:54.402 [DEBUG - Time-limited test] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:51:54.404 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.404 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.405 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.423 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:54.423 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:51:54.424 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1037525753(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@5e0db1cc -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1bfa2811 -> wakeup thread Time-limited test)) after lookup
05:51:54.425 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:51:54.427 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.427 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.427 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.443 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:54.444 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:51:54.444 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [23ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [25ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [25ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1037525753(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@5e0db1cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1bfa2811 -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131))) after lookup
05:51:54.445 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:51:54.446 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.446 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.446 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.464 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:54.464 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:51:54.465 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=63), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [23ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [25ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [25ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [42ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [44ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [44ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [61ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1037525753(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@5e0db1cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1bfa2811 -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198))) after lookup
05:51:54.465 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:51:54.467 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.467 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.467 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.503 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:54.504 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:51:54.505 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=103), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [23ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [25ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [25ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [42ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [44ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [44ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [61ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [63ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [65ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [66ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [98ms] refreshing cache from master, [98ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [100ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1037525753(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@5e0db1cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198) -> (continuation of Deferred@1011063608 after retry RPC@1776828331), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1bfa2811 -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198) -> (continuation of Deferred@1011063608 after retry RPC@1776828331))) after lookup
05:51:54.505 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:51:54.506 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.506 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.507 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.704 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:54.704 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:51:54.705 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=303), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [23ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [25ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [25ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [42ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [44ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [44ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [61ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [63ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [65ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [66ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [98ms] refreshing cache from master, [98ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [100ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [103ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [104ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [105ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [298ms] refreshing cache from master, [298ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [301ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1037525753(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@5e0db1cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198) -> (continuation of Deferred@1011063608 after retry RPC@1776828331) -> (continuation of Deferred@874340619 after retry RPC@962671484), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1bfa2811 -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198) -> (continuation of Deferred@1011063608 after retry RPC@1776828331) -> (continuation of Deferred@874340619 after retry RPC@962671484))) after lookup
05:51:54.706 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:51:54.708 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.708 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:54.708 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:51:55.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.722096 12481 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader abf53778adcf497787bbf7bac00509e2)
05:51:55.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.722365 12481 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:55.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.722751 12482 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader abf53778adcf497787bbf7bac00509e2)
05:51:55.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.722993 12482 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:55.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.724026 12481 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:55.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.724545 12482 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:55.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:55.725417 12042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:51:55.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:55.736291 12314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:51:55.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:55.736620 12042 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:51:55.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.736986 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160" is_pre_election: true
05:51:55.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.737315 12109 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 1.
05:51:55.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.737947 12317 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160; no voters: 
05:51:55.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.738433 12482 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 1 FOLLOWER]: Leader pre-election won for term 2
05:51:55.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.738574 12482 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 1 FOLLOWER]: Starting leader election (detected failure of leader abf53778adcf497787bbf7bac00509e2)
05:51:55.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.738692 12482 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 1 FOLLOWER]: Advancing to term 2
05:51:55.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:55.738605 12314 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:51:55.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.741080 12482 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:55.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.741948 12482 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 2 election: Requested vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:55.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.742617 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:51:55.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.742901 12109 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 1 FOLLOWER]: Advancing to term 2
05:51:55.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:55.743449 12314 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:51:55.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.744045 12381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba" is_pre_election: true
05:51:55.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.744266 12381 raft_consensus.cc:2379] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d2f80c10be94511bdabb93db3d6d160 in current term 2: Already voted for candidate 7c5d736667634e189b1c959e88b14aba in this term.
05:51:55.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.744917 12044 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160; no voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2
05:51:55.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.745721 12109 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 2.
05:51:55.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.745927 12481 raft_consensus.cc:2726] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
05:51:55.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.746182 12317 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160; no voters: abf53778adcf497787bbf7bac00509e2
05:51:55.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.746524 12482 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 2 FOLLOWER]: Leader election won for term 2
05:51:55.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.747072 12482 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 2 LEADER]: Becoming Leader. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Running, Role: LEADER
05:51:55.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.747608 12482 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:55.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:55.751382 11933 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba reported cstate change: term changed from 1 to 2, leader changed from abf53778adcf497787bbf7bac00509e2 (127.11.121.66) to 7c5d736667634e189b1c959e88b14aba (127.11.121.67). New cstate: current_term: 2 leader_uuid: "7c5d736667634e189b1c959e88b14aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:51:56.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:56.128692 12314 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba -> Peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Couldn't send request to peer abf53778adcf497787bbf7bac00509e2. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:51:56.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:56.141738 12109 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Refusing update from remote peer 7c5d736667634e189b1c959e88b14aba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
05:51:56.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:56.142644 12482 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
05:51:56.584 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:56.584 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:56.586 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2183), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [23ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [25ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [25ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [42ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [44ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [44ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [58ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [61ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [63ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [65ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [66ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [98ms] refreshing cache from master, [98ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [100ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [103ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [104ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [105ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [298ms] refreshing cache from master, [298ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [301ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [304ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [306ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [306ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [2177ms] refreshing cache from master, [2178ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [2181ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1037525753(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@5e0db1cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198) -> (continuation of Deferred@1011063608 after retry RPC@1776828331) -> (continuation of Deferred@874340619 after retry RPC@962671484) -> (continuation of Deferred@624389256 after retry RPC@360530305), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1bfa2811 -> wakeup thread Time-limited test -> (continuation of Deferred@1068591332 after retry RPC@1967220131) -> (continuation of Deferred@1687723302 after retry RPC@1913978198) -> (continuation of Deferred@1011063608 after retry RPC@1776828331) -> (continuation of Deferred@874340619 after retry RPC@962671484) -> (continuation of Deferred@624389256 after retry RPC@360530305))) after lookup
05:51:56.586 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:56.588 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] Successfully connected to peer
05:51:56.612 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xde6d5c3f, L:/127.0.0.1:36420 - R:/127.11.121.67:38133] using TOKEN/null
05:51:56.638 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.66:39283
05:51:56.640 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:51:56.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:56.641462 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:56.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:56.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:56.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:39283
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43205
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:56.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:51:56.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:56.987756 12510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:56.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:56.988097 12510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:56.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:56.988204 12510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:56.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:56.995672 12510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:56.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:56.995841 12510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.66
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.005120 12510 tablet_server_runner.cc:78] Tablet server non-default flags:
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:39283
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43205
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:57.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:57.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.006493 12510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:57.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.008421 12510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:57.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:57.021760 12517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:57.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:57.021809 12518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:57.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:57.021847 12520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:57.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.022879 12510 server_base.cc:1034] running on GCE node
05:51:57.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.023504 12510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:57.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.024602 12510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:57.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.025854 12510 hybrid_clock.cc:648] HybridClock initialized: now 1783489917025770 us; error 54 us; skew 500 ppm
05:51:57.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.026268 12510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:57.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.033556 12510 webserver.cc:460] Webserver started at http://127.11.121.66:43205/ using document root <none> and password file <none>
05:51:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.034330 12510 fs_manager.cc:359] Metadata directory not provided
05:51:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.034499 12510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:57.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.039902 12510 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.000s	sys 0.003s
05:51:57.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.043006 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:57.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.044409 12510 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:51:57.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.044600 12510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:57.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abf53778adcf497787bbf7bac00509e2"
05:51:57.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:57.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.045626 12510 fs_report.cc:383] FS layout report
05:51:57.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:57.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/data
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:57.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:57.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.068962 12510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:57.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.070048 12510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:57.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.070371 12510 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:57.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.071722 12510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:51:57.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.074198 12533 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:51:57.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.080252 12510 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:51:57.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.080422 12510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
05:51:57.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.080574 12510 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:51:57.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.083979 12510 ts_tablet_manager.cc:610] Registered 1 tablets
05:51:57.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.084107 12510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s	user 0.001s	sys 0.003s
05:51:57.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.084375 12533 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap starting.
05:51:57.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.104980 12533 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Log is configured to *not* fsync() on all Append() calls
05:51:57.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.120661 12533 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:51:57.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.121313 12533 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap complete.
05:51:57.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.122519 12533 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent bootstrapping tablet: real 0.038s	user 0.028s	sys 0.007s
05:51:57.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.132913 12533 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:57.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.134336 12533 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Initialized, Role: FOLLOWER
05:51:57.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.135004 12533 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:57.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.137470 12533 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.006s
05:51:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.146375 12510 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.66:39283
05:51:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.146432 12641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.66:39283 every 8 connection(s)
05:51:57.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.147785 12510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:51:57.150 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:51:57.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.149827 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 12510
05:51:57.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.180948 12643 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:51:57.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.181325 12643 heartbeater.cc:464] Registering TS with master...
05:51:57.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.182346 12643 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:51:57.183 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:51:57.184 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.67:38133
05:51:57.185 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:51:57.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.185549 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 12299
05:51:57.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.185871 11790 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:51:57.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.190054 12642 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:51:57.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.190219 12642 heartbeater.cc:464] Registering TS with master...
05:51:57.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.190629 12642 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:51:57.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.191165 12644 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:51:57.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.191278 12644 heartbeater.cc:464] Registering TS with master...
05:51:57.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.191715 12644 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:51:57.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.192430 11932 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:51:57.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.193778 11932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.66:41255
05:51:57.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.194963 11862 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:51:57.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.197448 12421 generic_service.cc:196] Checking for leaks (request via RPC)
05:51:57.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:57.293313 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 12299
05:51:57.311 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:51:57.311 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] handling channelInactive
05:51:57.311 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state READY due to: connection closed
05:51:57.312 [DEBUG - Time-limited test] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:57.314 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.314 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.315 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.323 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:57.324 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:57.324 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [11ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1494719502(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@33086a9a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37e82389 -> wakeup thread Time-limited test)) after lookup
05:51:57.325 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:57.326 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.326 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.326 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.343 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:57.344 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:57.344 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [11ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [14ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [14ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [30ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1494719502(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@33086a9a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37e82389 -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730))) after lookup
05:51:57.345 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:57.346 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.347 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.347 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.383 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:57.383 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:57.384 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=72), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [11ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [14ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [14ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [30ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [32ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [35ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [35ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [70ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1494719502(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@33086a9a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37e82389 -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041))) after lookup
05:51:57.385 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:57.387 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.387 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.387 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.464 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:57.464 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:57.465 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=152), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [11ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [14ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [14ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [30ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [32ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [35ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [35ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [70ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [73ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [75ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [75ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [151ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1494719502(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@33086a9a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37e82389 -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396))) after lookup
05:51:57.465 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:57.468 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.468 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.468 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.804 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:57.804 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:57.805 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=492), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [11ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [14ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [14ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [30ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [32ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [35ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [35ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [70ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [73ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [75ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [75ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [151ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [153ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [156ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [156ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [491ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1494719502(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@33086a9a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396) -> (continuation of Deferred@1640969782 after retry RPC@1187239680), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37e82389 -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396) -> (continuation of Deferred@1640969782 after retry RPC@1187239680))) after lookup
05:51:57.806 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:57.807 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.807 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.807 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.924 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:57.925 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:57.925 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=613), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [11ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [14ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [14ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [30ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [32ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [35ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [35ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [70ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [73ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [75ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [75ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [151ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [153ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [156ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [156ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [491ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [493ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [495ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [495ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [608ms] refreshing cache from master, [608ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [611ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1494719502(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@33086a9a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396) -> (continuation of Deferred@1640969782 after retry RPC@1187239680) -> (continuation of Deferred@216720647 after retry RPC@30051440), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37e82389 -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396) -> (continuation of Deferred@1640969782 after retry RPC@1187239680) -> (continuation of Deferred@216720647 after retry RPC@30051440))) after lookup
05:51:57.926 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:51:57.928 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.928 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:57.928 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:51:58.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.196750 12642 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:51:58.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.501752 12657 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:51:58.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.502054 12657 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:58.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.503737 12657 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:51:58.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:58.505322 12529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:51:58.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:58.507051 12529 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:51:58.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.511799 12109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160" is_pre_election: true
05:51:58.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.512774 12530 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160
05:51:58.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.513200 12657 raft_consensus.cc:2726] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
05:51:58.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.674290 12662 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:51:58.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.674477 12662 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:58.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.675074 12662 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:58.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:58.677866 12044 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:51:58.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.684455 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2" is_pre_election: true
05:51:58.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.684953 12596 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2f80c10be94511bdabb93db3d6d160 in term 1.
05:51:58.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.686169 12042 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:51:58.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.686504 12662 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Leader pre-election won for term 3
05:51:58.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.686663 12662 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:51:58.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.686800 12662 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 2 FOLLOWER]: Advancing to term 3
05:51:58.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.689359 12662 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:58.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.690071 12662 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 3 election: Requested vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:58.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.690533 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2"
05:51:58.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.690735 12596 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 1 FOLLOWER]: Advancing to term 3
05:51:58.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:58.692018 12044 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:51:58.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.695362 12596 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2f80c10be94511bdabb93db3d6d160 in term 3.
05:51:58.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.695827 12042 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:51:58.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.696233 12662 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Leader election won for term 3
05:51:58.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.696776 12662 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 LEADER]: Becoming Leader. State: Replica: 8d2f80c10be94511bdabb93db3d6d160, State: Running, Role: LEADER
05:51:58.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.697304 12662 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:58.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:58.701023 11932 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 reported cstate change: term changed from 2 to 3, leader changed from 7c5d736667634e189b1c959e88b14aba (127.11.121.67) to 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65). New cstate: current_term: 3 leader_uuid: "8d2f80c10be94511bdabb93db3d6d160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: HEALTHY } } }
05:51:59.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.290588 12596 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 3 FOLLOWER]: Refusing update from remote peer 8d2f80c10be94511bdabb93db3d6d160: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
05:51:59.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.291586 12681 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
05:51:59.305 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:51:59.306 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:51:59.307 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1994), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [11ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [14ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [14ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [30ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [32ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [35ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [35ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [70ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [73ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [75ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [75ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [148ms] refreshing cache from master, [148ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [151ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [153ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [156ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [156ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [491ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [493ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [495ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [495ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [608ms] refreshing cache from master, [608ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [611ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [614ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [616ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [616ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [1988ms] refreshing cache from master, [1988ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1991ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1494719502(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@33086a9a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396) -> (continuation of Deferred@1640969782 after retry RPC@1187239680) -> (continuation of Deferred@216720647 after retry RPC@30051440) -> (continuation of Deferred@524545040 after retry RPC@1245661828), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@37e82389 -> wakeup thread Time-limited test -> (continuation of Deferred@654561424 after retry RPC@1023880730) -> (continuation of Deferred@1712863406 after retry RPC@1651228041) -> (continuation of Deferred@419400140 after retry RPC@633288396) -> (continuation of Deferred@1640969782 after retry RPC@1187239680) -> (continuation of Deferred@216720647 after retry RPC@30051440) -> (continuation of Deferred@524545040 after retry RPC@1245661828))) after lookup
05:51:59.309 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:51:59.310 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] Successfully connected to peer
05:51:59.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.316565 12044 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 -> Peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Couldn't send request to peer 7c5d736667634e189b1c959e88b14aba. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:51:59.338 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x47d3cd42, L:/127.0.0.1:59270 - R:/127.11.121.65:46477] using TOKEN/null
05:51:59.365 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.67:38133
05:51:59.366 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:51:59.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.366789 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:59.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:51:59.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:51:59.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:59.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:59.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:59.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:51:59.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.698654 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:51:59.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.698945 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:51:59.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.699035 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:51:59.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.706611 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:51:59.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.706784 12696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.67
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.716244 12696 tablet_server_runner.cc:78] Tablet server non-default flags:
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:51:59.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:51:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.717694 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:51:59.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.719775 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:51:59.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.733168 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:59.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.733218 12702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:59.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:51:59.733248 12701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:51:59.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.734414 12696 server_base.cc:1034] running on GCE node
05:51:59.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.735044 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:51:59.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.736104 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:51:59.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.737397 12696 hybrid_clock.cc:648] HybridClock initialized: now 1783489919737301 us; error 39 us; skew 500 ppm
05:51:59.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.737804 12696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:51:59.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.740756 12696 webserver.cc:460] Webserver started at http://127.11.121.67:40053/ using document root <none> and password file <none>
05:51:59.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.741772 12696 fs_manager.cc:359] Metadata directory not provided
05:51:59.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.741992 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:51:59.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.747742 12696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
05:51:59.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.750835 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:51:59.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.752592 12696 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:51:59.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.752743 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:59.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7c5d736667634e189b1c959e88b14aba"
05:51:59.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:51:59.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.753754 12696 fs_report.cc:383] FS layout report
05:51:59.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/data
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:51:59.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:51:59.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.772785 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:51:59.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.773697 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:51:59.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.773969 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
05:51:59.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.774982 12696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:51:59.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.777107 12718 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:51:59.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.783269 12696 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:51:59.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.783373 12696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
05:51:59.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.783466 12696 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:51:59.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.786415 12696 ts_tablet_manager.cc:610] Registered 1 tablets
05:51:59.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.786490 12696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
05:51:59.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.787056 12718 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap starting.
05:51:59.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.814764 12718 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Log is configured to *not* fsync() on all Append() calls
05:51:59.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.845641 12718 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:51:59.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.846572 12718 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap complete.
05:51:59.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.848135 12718 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent bootstrapping tablet: real 0.061s	user 0.037s	sys 0.023s
05:51:59.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.861192 12718 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:59.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.862434 12718 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Initialized, Role: FOLLOWER
05:51:59.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.864969 12718 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:51:59.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.869410 12696 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.67:38133
05:51:59.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.869442 12826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.67:38133 every 8 connection(s)
05:51:59.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.871083 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:51:59.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.871244 12718 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent starting tablet: real 0.023s	user 0.013s	sys 0.003s
05:51:59.875 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:51:59.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.874646 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 12696
05:51:59.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.912240 12828 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:51:59.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.912590 12828 heartbeater.cc:464] Registering TS with master...
05:51:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.913776 12828 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:51:59.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.919118 11933 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:59.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.921491 11933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.67:49775
05:51:59.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.930871 12829 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:51:59.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.931080 12829 heartbeater.cc:464] Registering TS with master...
05:51:59.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.931632 12829 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:51:59.931 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:51:59.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.933689 11790 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:59.938 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.65:46477
05:51:59.938 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:51:59.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.939045 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 12027
05:51:59.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.944283 12832 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:51:59.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.944437 12832 heartbeater.cc:464] Registering TS with master...
05:51:59.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.944839 12832 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:51:59.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.946730 11862 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:51:59.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.953497 12781 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 2 FOLLOWER]: Advancing to term 3
05:51:59.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.955628 12149 generic_service.cc:196] Checking for leaks (request via RPC)
05:51:59.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:51:59.959538 12828 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:00.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:00.059484 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 12027
05:52:00.079 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:00.079 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] handling channelInactive
05:52:00.080 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state READY due to: connection closed
05:52:00.101 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:00.103 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.103 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.103 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.125 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:00.125 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:00.126 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=3, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [23ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [23ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@834868090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@1ac9a8e5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f23fa5 -> wakeup thread Time-limited test)) after lookup
05:52:00.127 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:00.128 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.128 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.128 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.164 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:00.165 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:00.165 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=5, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [23ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [23ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [48ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [48ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@834868090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@1ac9a8e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f23fa5 -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239))) after lookup
05:52:00.166 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:00.168 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.168 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.168 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.183 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:00.184 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:00.184 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=7, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=104), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [23ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [23ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [48ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [48ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [86ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [88ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [89ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [100ms] refreshing cache from master, [100ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [103ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@834868090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@1ac9a8e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f23fa5 -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414))) after lookup
05:52:00.185 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:00.186 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.186 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.186 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.263 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:00.263 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:00.264 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=9, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=184), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [23ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [23ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [48ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [48ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [86ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [88ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [89ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [100ms] refreshing cache from master, [100ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [103ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [105ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [106ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [106ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [179ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [182ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@834868090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@1ac9a8e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414) -> (continuation of Deferred@1750158631 after retry RPC@1965065982), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f23fa5 -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414) -> (continuation of Deferred@1750158631 after retry RPC@1965065982))) after lookup
05:52:00.265 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:00.267 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.267 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:00.267 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:01.124 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:01.124 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:01.124 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=11, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1044), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [23ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [23ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [48ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [48ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [86ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [88ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [89ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [100ms] refreshing cache from master, [100ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [103ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [105ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [106ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [106ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [179ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [182ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [185ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [187ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [187ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [1040ms] refreshing cache from master, [1040ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1043ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@834868090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@1ac9a8e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414) -> (continuation of Deferred@1750158631 after retry RPC@1965065982) -> (continuation of Deferred@2034826450 after retry RPC@145654387), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f23fa5 -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414) -> (continuation of Deferred@1750158631 after retry RPC@1965065982) -> (continuation of Deferred@2034826450 after retry RPC@145654387))) after lookup
05:52:01.125 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:01.127 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:01.127 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:01.127 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:01.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.370941 12848 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8d2f80c10be94511bdabb93db3d6d160)
05:52:01.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.371177 12848 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:01.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.372026 12848 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:01.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:01.374899 12530 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:01.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.378872 12781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba" is_pre_election: true
05:52:01.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.379882 12529 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160
05:52:01.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.380239 12848 raft_consensus.cc:2726] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
05:52:01.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.460884 12853 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8d2f80c10be94511bdabb93db3d6d160)
05:52:01.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.461200 12853 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:01.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.462705 12853 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:01.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:01.464046 12715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:01.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:01.465808 12715 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:01.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.470769 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2" is_pre_election: true
05:52:01.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.471176 12596 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 3.
05:52:01.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.471791 12712 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2; no voters: 8d2f80c10be94511bdabb93db3d6d160
05:52:01.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.472303 12853 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 3 FOLLOWER]: Leader pre-election won for term 4
05:52:01.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.472440 12853 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8d2f80c10be94511bdabb93db3d6d160)
05:52:01.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.472541 12853 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 3 FOLLOWER]: Advancing to term 4
05:52:01.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.474668 12853 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:01.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.475353 12853 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 4 election: Requested vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:01.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.475960 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2"
05:52:01.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.476229 12596 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 3 FOLLOWER]: Advancing to term 4
05:52:01.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:01.477303 12715 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.478534 12596 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 4.
05:52:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.478940 12712 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2; no voters: 8d2f80c10be94511bdabb93db3d6d160
05:52:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.479226 12853 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 4 FOLLOWER]: Leader election won for term 4
05:52:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.479686 12853 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 4 LEADER]: Becoming Leader. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Running, Role: LEADER
05:52:01.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.480278 12853 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:01.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.483872 11933 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba reported cstate change: term changed from 3 to 4, leader changed from 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65) to 7c5d736667634e189b1c959e88b14aba (127.11.121.67). New cstate: current_term: 4 leader_uuid: "7c5d736667634e189b1c959e88b14aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:52:01.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.920040 12596 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 4 FOLLOWER]: Refusing update from remote peer 7c5d736667634e189b1c959e88b14aba: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
05:52:01.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:01.921062 12853 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
05:52:01.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:01.930214 12715 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba -> Peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Couldn't send request to peer 8d2f80c10be94511bdabb93db3d6d160. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:04.199234 12715 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba -> Peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Couldn't send request to peer 8d2f80c10be94511bdabb93db3d6d160. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:52:05.183 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:05.184 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:05.184 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=13, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=5104), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [0ms] delaying RPC due to: Illegal state: connection is terminated, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [23ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [23ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [48ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [48ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [80ms] refreshing cache from master, [80ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [86ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [88ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [89ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [100ms] refreshing cache from master, [100ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [103ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [105ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [106ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [106ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [179ms] refreshing cache from master, [180ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [182ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [185ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [187ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [187ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [1040ms] refreshing cache from master, [1040ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1043ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [1045ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [1047ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [1047ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [5100ms] refreshing cache from master, [5100ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [5103ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@834868090(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@1ac9a8e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414) -> (continuation of Deferred@1750158631 after retry RPC@1965065982) -> (continuation of Deferred@2034826450 after retry RPC@145654387) -> (continuation of Deferred@1410755423 after retry RPC@122893015), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@59f23fa5 -> wakeup thread Time-limited test -> (continuation of Deferred@1262861063 after retry RPC@2044940239) -> (continuation of Deferred@1878919601 after retry RPC@201375414) -> (continuation of Deferred@1750158631 after retry RPC@1965065982) -> (continuation of Deferred@2034826450 after retry RPC@145654387) -> (continuation of Deferred@1410755423 after retry RPC@122893015))) after lookup
05:52:05.185 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:05.186 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] Successfully connected to peer
05:52:05.207 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xcc866f1b, L:/127.0.0.1:32958 - R:/127.11.121.67:38133] using TOKEN/null
05:52:05.231 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.65:46477
05:52:05.231 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.232359 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:52:05.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:46477
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34993
05:52:05.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:52:05.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:05.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:05.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:05.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.576851 12889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.577191 12889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.577282 12889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:05.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.585280 12889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:05.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.585474 12889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.65
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.596158 12889 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:46477
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34993
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:05.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:05.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.597681 12889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:05.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.599524 12889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:05.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.614199 12894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:05.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.614138 12897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:05.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:05.614101 12895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:05.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.615293 12889 server_base.cc:1034] running on GCE node
05:52:05.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.616056 12889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:05.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.617056 12889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:05.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.618367 12889 hybrid_clock.cc:648] HybridClock initialized: now 1783489925618279 us; error 50 us; skew 500 ppm
05:52:05.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.618727 12889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:05.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.621187 12889 webserver.cc:460] Webserver started at http://127.11.121.65:34993/ using document root <none> and password file <none>
05:52:05.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.621918 12889 fs_manager.cc:359] Metadata directory not provided
05:52:05.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.622085 12889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:05.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.627377 12889 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
05:52:05.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.630463 12903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:05.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.631839 12889 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.632061 12889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:51 on dist-test-slave-w6r9"
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.633078 12889 fs_report.cc:383] FS layout report
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/data
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:05.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:05.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.646396 12889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:05.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.647337 12889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:05.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.647670 12889 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:05.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.648785 12889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:05.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.651109 12910 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:05.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.656917 12889 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:05.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.657029 12889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
05:52:05.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.657121 12889 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:05.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.660461 12889 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:05.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.660565 12889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
05:52:05.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.661123 12910 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap starting.
05:52:05.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.693404 12910 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Log is configured to *not* fsync() on all Append() calls
05:52:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.716943 12910 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.717464 12910 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap complete.
05:52:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.718629 12910 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent bootstrapping tablet: real 0.058s	user 0.046s	sys 0.009s
05:52:05.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.726284 12889 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.65:46477
05:52:05.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.726389 13017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.65:46477 every 8 connection(s)
05:52:05.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.728008 12889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:05.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.729725 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 12889
05:52:05.730 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:05.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.729322 12910 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:05.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.730868 12910 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f80c10be94511bdabb93db3d6d160, State: Initialized, Role: FOLLOWER
05:52:05.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.739769 12910 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:05.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.747519 12910 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.010s
05:52:05.760 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:05.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.778565 13018 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:05.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.778970 13018 heartbeater.cc:464] Registering TS with master...
05:52:05.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.780110 13018 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.783159 11932 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.784783 11932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.65:43863
05:52:05.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.788239 13018 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:05.795 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.67:38133
05:52:05.795 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:52:05.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.796240 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 12696
05:52:05.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.818589 13024 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:05.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.818744 13024 heartbeater.cc:464] Registering TS with master...
05:52:05.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.819105 13024 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.820652 11862 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:05.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.821677 13019 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:05.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.821811 13019 heartbeater.cc:464] Registering TS with master...
05:52:05.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.822053 13019 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:05.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.822093 12821 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:05.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.823549 11791 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:05.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:05.936940 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 12696
05:52:05.955 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] handling channelInactive
05:52:05.955 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:05.956 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state READY due to: connection closed
05:52:05.956 [DEBUG - Time-limited test] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:05.958 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:05.958 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:05.959 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:05.984 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:05.984 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:05.984 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [27ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@182998428(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@7b93c102 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33f22960 -> wakeup thread Time-limited test)) after lookup
05:52:05.985 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:05.987 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:05.987 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:05.988 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.003 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:06.003 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:06.003 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [27ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [31ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [32ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [46ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@182998428(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@7b93c102 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33f22960 -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277))) after lookup
05:52:06.004 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:06.005 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.005 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.006 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.043 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:06.043 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:06.044 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=87), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [27ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [31ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [32ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [46ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [48ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [50ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [50ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [86ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@182998428(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@7b93c102 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33f22960 -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822))) after lookup
05:52:06.044 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:06.048 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.048 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.049 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.084 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:06.084 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:06.084 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=128), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [27ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [31ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [32ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [46ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [48ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [50ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [50ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [86ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [88ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [93ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [93ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [124ms] refreshing cache from master, [124ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [127ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@182998428(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@7b93c102 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822) -> (continuation of Deferred@1184442667 after retry RPC@361872266), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33f22960 -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822) -> (continuation of Deferred@1184442667 after retry RPC@361872266))) after lookup
05:52:06.085 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:06.086 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.086 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.086 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.503 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:06.503 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:06.504 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=548), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [27ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [31ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [32ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [46ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [48ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [50ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [50ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [86ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [88ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [93ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [93ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [124ms] refreshing cache from master, [124ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [127ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [129ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [130ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [130ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [544ms] refreshing cache from master, [544ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [547ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@182998428(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@7b93c102 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822) -> (continuation of Deferred@1184442667 after retry RPC@361872266) -> (continuation of Deferred@332672050 after retry RPC@1219281909), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33f22960 -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822) -> (continuation of Deferred@1184442667 after retry RPC@361872266) -> (continuation of Deferred@332672050 after retry RPC@1219281909))) after lookup
05:52:06.505 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:06.507 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.507 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.507 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:06.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:06.980829 13033 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:52:06.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:06.981228 13033 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:06.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:06.982936 13033 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:06.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:06.985384 12906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:06.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:06.986944 12906 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:06.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:06.992100 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2" is_pre_election: true
05:52:06.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:06.993000 12904 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160; no voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2
05:52:06.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:06.993408 13033 raft_consensus.cc:2726] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
05:52:07.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.438042 13038 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:52:07.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.438328 13038 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:07.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.439234 13038 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:07.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:07.441478 12529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111) [suppressed 1 similar messages]
05:52:07.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:07.443146 12529 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:07.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.448777 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160" is_pre_election: true
05:52:07.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.449321 12973 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 3.
05:52:07.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.450624 12530 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:52:07.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.451004 13038 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 4 FOLLOWER]: Leader pre-election won for term 5
05:52:07.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.451145 13038 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:52:07.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.451274 13038 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 4 FOLLOWER]: Advancing to term 5
05:52:07.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.453464 13038 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:07.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.454118 13038 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 5 election: Requested vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:07.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.454799 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:52:07.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.455042 12973 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 3 FOLLOWER]: Advancing to term 5
05:52:07.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:07.455611 12529 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:07.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.459102 12973 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 5.
05:52:07.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.459573 12530 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:52:07.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.459968 13038 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 5 FOLLOWER]: Leader election won for term 5
05:52:07.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.460461 13038 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 5 LEADER]: Becoming Leader. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Running, Role: LEADER
05:52:07.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.460994 13038 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:07.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.464351 11932 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 reported cstate change: term changed from 4 to 5, leader changed from 7c5d736667634e189b1c959e88b14aba (127.11.121.67) to abf53778adcf497787bbf7bac00509e2 (127.11.121.66). New cstate: current_term: 5 leader_uuid: "abf53778adcf497787bbf7bac00509e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:52:07.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:07.852190 12529 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 -> Peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Couldn't send request to peer 7c5d736667634e189b1c959e88b14aba. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:07.882 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:07.883 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:07.883 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1927), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [27ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [31ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [32ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [46ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [48ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [50ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [50ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [84ms] refreshing cache from master, [84ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [86ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [88ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [93ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [93ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [124ms] refreshing cache from master, [124ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [127ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [129ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [130ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [130ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [544ms] refreshing cache from master, [544ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [547ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [549ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [551ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [551ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [1923ms] refreshing cache from master, [1923ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1926ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@182998428(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@7b93c102 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822) -> (continuation of Deferred@1184442667 after retry RPC@361872266) -> (continuation of Deferred@332672050 after retry RPC@1219281909) -> (continuation of Deferred@1326103726 after retry RPC@1628929489), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@33f22960 -> wakeup thread Time-limited test -> (continuation of Deferred@311801854 after retry RPC@814626277) -> (continuation of Deferred@1621358196 after retry RPC@628167822) -> (continuation of Deferred@1184442667 after retry RPC@361872266) -> (continuation of Deferred@332672050 after retry RPC@1219281909) -> (continuation of Deferred@1326103726 after retry RPC@1628929489))) after lookup
05:52:07.884 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:07.885 [DEBUG - kudu-nio-0] (Connection.java:757) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] Successfully connected to peer
05:52:07.906 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xb1dc8ed5, L:/127.0.0.1:43062 - R:/127.11.121.66:39283] using TOKEN/null
05:52:07.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.912165 12973 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 5 FOLLOWER]: Refusing update from remote peer abf53778adcf497787bbf7bac00509e2: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
05:52:07.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.912987 13038 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
05:52:07.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.923436 13057 mvcc.cc:205] Tried to move back new op lower bound from 7305174744719196160 to 7305174742883733504. Current Snapshot: MvccSnapshot[applied={T|T < 7305174744719196160}]
05:52:07.943 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.67:38133
05:52:07.944 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:07.944839 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:08.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.285811 13066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:08.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.286167 13066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:08.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.286284 13066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.294103 13066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.294344 13066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.67
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.303714 13066 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:08.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:08.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.305469 13066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:08.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.307423 13066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:08.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.321480 13072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:08.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.321404 13073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:08.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:08.321522 13075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:08.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.322918 13066 server_base.cc:1034] running on GCE node
05:52:08.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.323573 13066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:08.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.324563 13066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:08.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.325774 13066 hybrid_clock.cc:648] HybridClock initialized: now 1783489928325705 us; error 43 us; skew 500 ppm
05:52:08.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.326049 13066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:08.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.328474 13066 webserver.cc:460] Webserver started at http://127.11.121.67:40053/ using document root <none> and password file <none>
05:52:08.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.329250 13066 fs_manager.cc:359] Metadata directory not provided
05:52:08.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.329360 13066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:08.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.335435 13066 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
05:52:08.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.339385 13081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:08.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.341231 13066 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
05:52:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.341411 13066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7c5d736667634e189b1c959e88b14aba"
05:52:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.342877 13066 fs_report.cc:383] FS layout report
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/data
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:08.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:08.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.372325 13066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:08.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.373385 13066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:08.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.373651 13066 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.374694 13066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.377220 13088 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.383252 13066 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.383381 13066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
05:52:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.383482 13066 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.386721 13066 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.386817 13066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
05:52:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.387259 13088 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap starting.
05:52:08.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.413058 13088 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Log is configured to *not* fsync() on all Append() calls
05:52:08.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.435964 13088 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:08.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.436852 13088 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap complete.
05:52:08.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.438597 13088 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent bootstrapping tablet: real 0.052s	user 0.030s	sys 0.019s
05:52:08.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.444900 13066 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.67:38133
05:52:08.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.444957 13195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.67:38133 every 8 connection(s)
05:52:08.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.446295 13066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:08.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.448338 13088 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:08.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.449935 13088 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Initialized, Role: FOLLOWER
05:52:08.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.450954 13088 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:08.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.455718 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 13066
05:52:08.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:08.519 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:08.520 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.66:39283
05:52:08.520 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:52:08.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.520784 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 12510
05:52:08.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.535449 13198 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:08.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.535851 13198 heartbeater.cc:464] Registering TS with master...
05:52:08.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.537050 13198 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:08.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.540105 13197 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:08.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.540246 13197 heartbeater.cc:464] Registering TS with master...
05:52:08.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.540254 11862 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:08.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.540715 13197 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:08.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.542407 11791 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:08.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.543426 13088 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent starting tablet: real 0.105s	user 0.032s	sys 0.066s
05:52:08.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.547757 13196 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:08.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.547891 13196 heartbeater.cc:464] Registering TS with master...
05:52:08.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.548357 13196 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:08.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.549574 11932 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:08.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.551108 11932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.67:41529
05:52:08.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.552457 12637 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:08.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.554015 13196 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:08.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:08.651641 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 12510
05:52:08.670 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:08.670 [DEBUG - kudu-nio-0] (Connection.java:239) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] handling channelInactive
05:52:08.670 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state READY due to: connection closed
05:52:08.670 [DEBUG - Time-limited test] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:08.673 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.673 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.673 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.682 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:08.683 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:08.683 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test)) after lookup
05:52:08.684 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:08.685 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.685 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.685 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.703 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:08.703 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:08.704 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [15ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [15ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [33ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991))) after lookup
05:52:08.704 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:08.706 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.706 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.707 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.723 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:08.723 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:08.723 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [15ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [15ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [33ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [34ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [37ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [37ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [49ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [52ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137))) after lookup
05:52:08.725 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:08.726 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.726 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.726 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.763 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:08.763 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:08.764 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=93), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [15ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [15ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [33ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [34ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [37ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [37ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [49ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [52ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [54ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [56ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [56ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [92ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583))) after lookup
05:52:08.764 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:08.766 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.766 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:08.766 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.223 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:09.223 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:09.224 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=553), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [15ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [15ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [33ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [34ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [37ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [37ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [49ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [52ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [54ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [56ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [56ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [92ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [94ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [96ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [97ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [549ms] refreshing cache from master, [550ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [552ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289))) after lookup
05:52:09.225 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:09.226 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.226 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.226 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.324 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:09.324 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:09.325 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=654), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [15ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [15ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [33ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [34ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [37ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [37ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [49ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [52ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [54ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [56ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [56ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [92ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [94ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [96ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [97ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [549ms] refreshing cache from master, [550ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [552ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [554ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [556ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [556ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [650ms] refreshing cache from master, [650ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [653ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289) -> (continuation of Deferred@1180694583 after retry RPC@975560639), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289) -> (continuation of Deferred@1180694583 after retry RPC@975560639))) after lookup
05:52:09.326 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:09.327 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.327 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.328 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.583 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:09.583 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:09.604 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=14, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=913), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [15ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [15ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [33ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [34ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [37ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [37ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [49ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [52ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [54ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [56ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [56ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [92ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [94ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [96ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [97ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [549ms] refreshing cache from master, [550ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [552ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [554ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [556ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [556ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [650ms] refreshing cache from master, [650ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [653ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [655ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [658ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [658ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [909ms] refreshing cache from master, [910ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [913ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289) -> (continuation of Deferred@1180694583 after retry RPC@975560639) -> (continuation of Deferred@118308886 after retry RPC@323071855), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289) -> (continuation of Deferred@1180694583 after retry RPC@975560639) -> (continuation of Deferred@118308886 after retry RPC@323071855))) after lookup
05:52:09.605 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:09.607 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.607 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.607 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:09.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.978258 13211 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader abf53778adcf497787bbf7bac00509e2)
05:52:09.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.978447 13211 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:09.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.979147 13211 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:09.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:09.981356 12904 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:52:09.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.987764 13151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba" is_pre_election: true
05:52:09.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.988384 13151 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2f80c10be94511bdabb93db3d6d160 in term 4.
05:52:09.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.989471 12906 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160; no voters: abf53778adcf497787bbf7bac00509e2
05:52:09.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.989840 13211 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 5 FOLLOWER]: Leader pre-election won for term 6
05:52:09.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.989949 13211 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 5 FOLLOWER]: Starting leader election (detected failure of leader abf53778adcf497787bbf7bac00509e2)
05:52:09.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.990016 13211 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 5 FOLLOWER]: Advancing to term 6
05:52:09.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.992179 13211 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:09.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.992811 13211 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 6 election: Requested vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:09.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.993398 13151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba"
05:52:09.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.993651 13151 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 4 FOLLOWER]: Advancing to term 6
05:52:09.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:09.994395 12904 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:52:09.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.997761 13151 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2f80c10be94511bdabb93db3d6d160 in term 6.
05:52:09.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.998210 12906 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160; no voters: abf53778adcf497787bbf7bac00509e2
05:52:09.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.998533 13211 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 6 FOLLOWER]: Leader election won for term 6
05:52:09.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.999073 13211 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 6 LEADER]: Becoming Leader. State: Replica: 8d2f80c10be94511bdabb93db3d6d160, State: Running, Role: LEADER
05:52:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:09.999780 13211 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:10.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.003515 11932 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 reported cstate change: term changed from 5 to 6, leader changed from abf53778adcf497787bbf7bac00509e2 (127.11.121.66) to 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65). New cstate: current_term: 6 leader_uuid: "8d2f80c10be94511bdabb93db3d6d160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: HEALTHY } } }
05:52:10.083 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:10.083 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:10.084 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=16, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1413), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [3ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [12ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [13ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [15ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [15ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [30ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [33ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [34ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [37ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [37ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [49ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [52ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [54ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [56ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [56ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [92ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [94ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [96ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [97ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [549ms] refreshing cache from master, [550ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [552ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [554ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [556ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [556ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [650ms] refreshing cache from master, [650ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [653ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [655ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [658ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [658ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [909ms] refreshing cache from master, [910ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [913ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [935ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [937ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [938ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [1409ms] refreshing cache from master, [1410ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1412ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@1830316740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3edfb27c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289) -> (continuation of Deferred@1180694583 after retry RPC@975560639) -> (continuation of Deferred@118308886 after retry RPC@323071855) -> (continuation of Deferred@1066559219 after retry RPC@176677012), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74b4c48f -> wakeup thread Time-limited test -> (continuation of Deferred@2111947621 after retry RPC@226689991) -> (continuation of Deferred@1025202790 after retry RPC@672398137) -> (continuation of Deferred@1896957373 after retry RPC@1960362583) -> (continuation of Deferred@1469393510 after retry RPC@1701679289) -> (continuation of Deferred@1180694583 after retry RPC@975560639) -> (continuation of Deferred@118308886 after retry RPC@323071855) -> (continuation of Deferred@1066559219 after retry RPC@176677012))) after lookup
05:52:10.085 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:10.087 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] Successfully connected to peer
05:52:10.109 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x61dccc6c, L:/127.0.0.1:37392 - R:/127.11.121.65:46477] using TOKEN/null
05:52:10.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.115000 13151 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 6 FOLLOWER]: Refusing update from remote peer 8d2f80c10be94511bdabb93db3d6d160: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
05:52:10.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.115187 12904 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 -> Peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Couldn't send request to peer abf53778adcf497787bbf7bac00509e2. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:10.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.116113 13211 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
05:52:10.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.127588 13229 mvcc.cc:205] Tried to move back new op lower bound from 7305174753741488128 to 7305174753283305472. Current Snapshot: MvccSnapshot[applied={T|T < 7305174753741488128}]
05:52:10.147 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.66:39283
05:52:10.149 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.150106 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:39283
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43205
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:10.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.498909 13238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.499202 13238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.499289 13238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:10.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.506947 13238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:10.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.507174 13238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.66
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.516996 13238 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:39283
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43205
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:10.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:10.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:10.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:10.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:10.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:10.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:10.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:10.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.518358 13238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:10.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.520263 13238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:10.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.534282 13246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:10.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.534340 13244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:10.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:10.534341 13243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:10.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.537401 13238 server_base.cc:1034] running on GCE node
05:52:10.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.538070 13238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:10.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.539130 13238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:10.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.540377 13238 hybrid_clock.cc:648] HybridClock initialized: now 1783489930540312 us; error 34 us; skew 500 ppm
05:52:10.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.540764 13238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:10.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.546521 13238 webserver.cc:460] Webserver started at http://127.11.121.66:43205/ using document root <none> and password file <none>
05:52:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.547303 13238 fs_manager.cc:359] Metadata directory not provided
05:52:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.547459 13238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:10.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.552940 13238 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
05:52:10.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.556011 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:10.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.557349 13238 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
05:52:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.557535 13238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abf53778adcf497787bbf7bac00509e2"
05:52:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:52:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.558602 13238 fs_report.cc:383] FS layout report
05:52:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/data
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:10.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.589175 13238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:10.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.590094 13238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:10.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.590436 13238 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:10.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.591423 13238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:10.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.593658 13259 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:10.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.599406 13238 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:10.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.599521 13238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
05:52:10.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.599617 13238 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:10.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.602756 13238 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:10.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.602833 13238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
05:52:10.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.603334 13259 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap starting.
05:52:10.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.633744 13259 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Log is configured to *not* fsync() on all Append() calls
05:52:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.641557 13366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.66:39283 every 8 connection(s)
05:52:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.641522 13238 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.66:39283
05:52:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.643275 13238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:52:10.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.647341 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 13238
05:52:10.648 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:10.654 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
05:52:10.656 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
05:52:10.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.679265 13369 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:10.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.679656 13369 heartbeater.cc:464] Registering TS with master...
05:52:10.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.680756 13369 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:10.682 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:10.684 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.65:46477
05:52:10.684 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:52:10.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.685024 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 12889
05:52:10.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.685209 13368 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:10.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.685374 13368 heartbeater.cc:464] Registering TS with master...
05:52:10.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.686085 13368 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:10.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.686738 11862 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:52:10.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.692412 13367 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:10.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.692576 13367 heartbeater.cc:464] Registering TS with master...
05:52:10.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.693097 13367 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:10.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.694060 11791 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:52:10.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.695608 11932 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:52:10.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.697136 11932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.66:45805
05:52:10.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.697641 13259 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:10.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.698319 13259 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap complete.
05:52:10.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.699870 13259 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent bootstrapping tablet: real 0.097s	user 0.040s	sys 0.023s
05:52:10.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.707684 13259 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:10.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.709077 13259 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Initialized, Role: FOLLOWER
05:52:10.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.709486 13013 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:10.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.709901 13259 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:10.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.711154 13367 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:10.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.711774 13259 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.001s
05:52:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:10.813868 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 12889
05:52:10.832 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] handling channelInactive
05:52:10.833 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state READY due to: connection closed
05:52:10.833 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:10.833 [DEBUG - Time-limited test] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:10.835 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.835 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.835 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.842 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:10.843 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:10.843 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [9ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@681031920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@25543e9c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4e930e7c -> wakeup thread Time-limited test)) after lookup
05:52:10.844 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:10.845 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.845 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.846 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.862 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:10.863 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:10.863 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [9ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [10ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [13ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [13ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [29ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@681031920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@25543e9c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4e930e7c -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269))) after lookup
05:52:10.864 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:10.865 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.865 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.865 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.883 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:10.884 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:10.884 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [9ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [10ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [13ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [13ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [29ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [31ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [32ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [50ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@681031920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@25543e9c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4e930e7c -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391))) after lookup
05:52:10.885 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:10.886 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.886 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.886 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.983 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:10.983 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:10.983 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=150), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [9ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [10ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [13ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [13ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [29ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [31ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [32ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [50ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [51ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [53ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [54ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [149ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@681031920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@25543e9c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4e930e7c -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326))) after lookup
05:52:10.984 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:10.985 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.985 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:10.985 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:11.023 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:11.023 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:11.024 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [9ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [10ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [13ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [13ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [29ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [31ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [32ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [50ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [51ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [53ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [54ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [149ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [151ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [152ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [152ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [190ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@681031920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@25543e9c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326) -> (continuation of Deferred@1136039441 after retry RPC@1185667273), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4e930e7c -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326) -> (continuation of Deferred@1136039441 after retry RPC@1185667273))) after lookup
05:52:11.024 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:11.026 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:11.026 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:11.027 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:11.863 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:11.863 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:11.864 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1030), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [9ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [10ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [13ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [13ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [29ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [31ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [32ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [50ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [51ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [53ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [54ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [149ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [151ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [152ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [152ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [190ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [191ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [194ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [194ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [1026ms] refreshing cache from master, [1027ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1030ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@681031920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@25543e9c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326) -> (continuation of Deferred@1136039441 after retry RPC@1185667273) -> (continuation of Deferred@446997852 after retry RPC@47274833), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4e930e7c -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326) -> (continuation of Deferred@1136039441 after retry RPC@1185667273) -> (continuation of Deferred@446997852 after retry RPC@47274833))) after lookup
05:52:11.864 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:11.865 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:11.865 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:11.866 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:12.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.164776 13382 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8d2f80c10be94511bdabb93db3d6d160)
05:52:12.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.165009 13382 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:12.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.167008 13382 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:12.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:12.170671 13085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:12.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:12.172938 13085 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:12.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.180238 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2" is_pre_election: true
05:52:12.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.180752 13321 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 5.
05:52:12.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.181809 13082 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2; no voters: 8d2f80c10be94511bdabb93db3d6d160
05:52:12.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.182250 13382 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 6 FOLLOWER]: Leader pre-election won for term 7
05:52:12.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.182366 13382 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8d2f80c10be94511bdabb93db3d6d160)
05:52:12.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.182428 13382 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 6 FOLLOWER]: Advancing to term 7
05:52:12.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.184492 13382 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:12.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.185170 13382 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 7 election: Requested vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:12.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.185721 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2"
05:52:12.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.185979 13321 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 5 FOLLOWER]: Advancing to term 7
05:52:12.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:12.186794 13085 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:12.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.190555 13321 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 7.
05:52:12.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.191040 13082 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2; no voters: 8d2f80c10be94511bdabb93db3d6d160
05:52:12.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.191344 13382 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 7 FOLLOWER]: Leader election won for term 7
05:52:12.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.191896 13382 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 7 LEADER]: Becoming Leader. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Running, Role: LEADER
05:52:12.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.192492 13382 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:12.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.196198 11932 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba reported cstate change: term changed from 6 to 7, leader changed from 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65) to 7c5d736667634e189b1c959e88b14aba (127.11.121.67). New cstate: current_term: 7 leader_uuid: "7c5d736667634e189b1c959e88b14aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:52:12.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.594815 13321 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 7 FOLLOWER]: Refusing update from remote peer 7c5d736667634e189b1c959e88b14aba: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
05:52:12.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:12.595709 13382 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
05:52:12.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:12.613776 13085 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba -> Peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Couldn't send request to peer 8d2f80c10be94511bdabb93db3d6d160. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:14.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:14.990430 13085 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba -> Peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Couldn't send request to peer 8d2f80c10be94511bdabb93db3d6d160. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:52:15.822 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:15.823 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:15.823 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4989), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [2ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [9ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [10ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [13ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [13ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [29ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [31ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [32ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [32ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [50ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [51ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [53ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [54ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [147ms] refreshing cache from master, [147ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [149ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [151ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [152ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [152ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [190ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [191ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [194ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [194ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [1026ms] refreshing cache from master, [1027ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1030ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [1031ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [1032ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [1033ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [4986ms] refreshing cache from master, [4986ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [4989ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@681031920(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@25543e9c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326) -> (continuation of Deferred@1136039441 after retry RPC@1185667273) -> (continuation of Deferred@446997852 after retry RPC@47274833) -> (continuation of Deferred@983385302 after retry RPC@1741358288), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4e930e7c -> wakeup thread Time-limited test -> (continuation of Deferred@922814711 after retry RPC@1639387269) -> (continuation of Deferred@1134615511 after retry RPC@2069407391) -> (continuation of Deferred@935150816 after retry RPC@158298326) -> (continuation of Deferred@1136039441 after retry RPC@1185667273) -> (continuation of Deferred@446997852 after retry RPC@47274833) -> (continuation of Deferred@983385302 after retry RPC@1741358288))) after lookup
05:52:15.824 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:15.825 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] Successfully connected to peer
05:52:15.849 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x3a41cce2, L:/127.0.0.1:34580 - R:/127.11.121.67:38133] using TOKEN/null
05:52:15.874 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.65:46477
05:52:15.874 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:52:15.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:15.875660 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:15.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:15.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:46477
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34993
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:15.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:16.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.216572 13420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:16.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.216871 13420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:16.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.216952 13420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:16.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.224344 13420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:16.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.224522 13420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.65
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.233940 13420 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:46477
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34993
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:16.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:16.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.235262 13420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:16.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.237062 13420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.250686 13425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.250686 13426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:16.250723 13428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.251274 13420 server_base.cc:1034] running on GCE node
05:52:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.252063 13420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.253108 13420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:16.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.254385 13420 hybrid_clock.cc:648] HybridClock initialized: now 1783489936254293 us; error 59 us; skew 500 ppm
05:52:16.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.254765 13420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:16.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.261756 13420 webserver.cc:460] Webserver started at http://127.11.121.65:34993/ using document root <none> and password file <none>
05:52:16.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.262639 13420 fs_manager.cc:359] Metadata directory not provided
05:52:16.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.262797 13420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:16.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.268355 13420 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
05:52:16.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.271407 13434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:16.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.273092 13420 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.273308 13420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:51 on dist-test-slave-w6r9"
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.274498 13420 fs_report.cc:383] FS layout report
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:16.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/data
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:16.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.318354 13420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:16.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.319344 13420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:16.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.319697 13420 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:16.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.320832 13420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:16.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.322944 13441 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:16.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.328727 13420 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:16.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.328881 13420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
05:52:16.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.329010 13420 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:16.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.332197 13420 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:16.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.332305 13420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
05:52:16.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.332684 13441 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap starting.
05:52:16.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.360239 13441 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Log is configured to *not* fsync() on all Append() calls
05:52:16.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.387671 13420 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.65:46477
05:52:16.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.387693 13548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.65:46477 every 8 connection(s)
05:52:16.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.389150 13420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:16.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.393244 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 13420
05:52:16.394 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:16.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.393388 13441 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:16.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.411013 13441 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap complete.
05:52:16.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.412616 13441 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent bootstrapping tablet: real 0.080s	user 0.047s	sys 0.015s
05:52:16.424 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:16.424 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.67:38133
05:52:16.425 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:52:16.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.426038 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 13066
05:52:16.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.438354 13550 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:16.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.438721 13550 heartbeater.cc:464] Registering TS with master...
05:52:16.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.439800 13550 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:16.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.444609 11791 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:16.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.446359 13191 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:16.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.448550 13549 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:16.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.448695 13549 heartbeater.cc:464] Registering TS with master...
05:52:16.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.449167 13549 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:16.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.450681 11933 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:16.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.452559 11933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.65:51037
05:52:16.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.452093 13441 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:16.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.453483 13441 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f80c10be94511bdabb93db3d6d160, State: Initialized, Role: FOLLOWER
05:52:16.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.454123 13441 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:16.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.456910 13549 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:16.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.458117 13551 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:16.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.459525 13551 heartbeater.cc:464] Registering TS with master...
05:52:16.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.459932 13551 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:16.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.458704 13441 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent starting tablet: real 0.046s	user 0.010s	sys 0.004s
05:52:16.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.461242 11861 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:16.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:16.559067 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 13066
05:52:16.578 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:16.578 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] handling channelInactive
05:52:16.578 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state READY due to: connection closed
05:52:16.578 [DEBUG - Time-limited test] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:16.580 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.580 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.581 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.603 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:16.603 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:16.603 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [24ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@353492820(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@61dffc3a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ff843ba -> wakeup thread Time-limited test)) after lookup
05:52:16.604 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:16.605 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.605 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.606 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.623 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:16.623 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:16.624 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [24ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [26ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [28ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [28ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@353492820(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@61dffc3a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ff843ba -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095))) after lookup
05:52:16.624 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:16.625 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.625 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.626 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.662 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:16.662 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:16.663 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=84), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [24ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [26ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [28ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [28ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [48ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@353492820(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@61dffc3a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ff843ba -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957))) after lookup
05:52:16.664 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:16.665 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.665 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.665 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.803 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:16.803 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:16.804 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=225), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [24ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [26ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [28ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [28ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [48ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [85ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [87ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [87ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [222ms] refreshing cache from master, [222ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [224ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@353492820(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@61dffc3a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ff843ba -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117))) after lookup
05:52:16.804 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:16.806 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.806 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.806 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.842 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:16.843 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:16.843 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=265), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [24ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [26ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [28ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [28ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [48ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [85ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [87ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [87ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [222ms] refreshing cache from master, [222ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [224ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [226ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [228ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [228ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [261ms] refreshing cache from master, [262ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [264ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@353492820(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@61dffc3a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117) -> (continuation of Deferred@1406928045 after retry RPC@1009215526), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ff843ba -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117) -> (continuation of Deferred@1406928045 after retry RPC@1009215526))) after lookup
05:52:16.844 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:16.845 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.845 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:16.845 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:17.383 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:17.383 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:17.385 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=807), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [24ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [26ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [28ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [28ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [48ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [85ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [87ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [87ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [222ms] refreshing cache from master, [222ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [224ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [226ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [228ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [228ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [261ms] refreshing cache from master, [262ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [264ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [265ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [267ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [268ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [802ms] refreshing cache from master, [802ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [804ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@353492820(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@61dffc3a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117) -> (continuation of Deferred@1406928045 after retry RPC@1009215526) -> (continuation of Deferred@488642458 after retry RPC@440478272), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ff843ba -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117) -> (continuation of Deferred@1406928045 after retry RPC@1009215526) -> (continuation of Deferred@488642458 after retry RPC@440478272))) after lookup
05:52:17.386 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:17.387 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:17.387 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:17.388 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:17.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.835721 13563 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:52:17.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.835942 13563 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:17.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.837324 13563 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:17.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:17.841264 13255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:17.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:17.843008 13255 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:17.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.848851 13504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160" is_pre_election: true
05:52:17.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.849428 13504 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 6.
05:52:17.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.850595 13256 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:52:17.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.851069 13563 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 7 FOLLOWER]: Leader pre-election won for term 8
05:52:17.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.851187 13563 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:52:17.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.851267 13563 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 7 FOLLOWER]: Advancing to term 8
05:52:17.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.853255 13563 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:17.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.853812 13563 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 8 election: Requested vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.854327 13504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:52:17.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.854525 13504 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 6 FOLLOWER]: Advancing to term 8
05:52:17.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:17.855338 13255 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:17.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.858596 13504 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 8.
05:52:17.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.859076 13256 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:52:17.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.859426 13563 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Leader election won for term 8
05:52:17.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.859881 13563 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 LEADER]: Becoming Leader. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Running, Role: LEADER
05:52:17.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.860431 13563 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:17.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:17.864248 11933 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 reported cstate change: term changed from 7 to 8, leader changed from 7c5d736667634e189b1c959e88b14aba (127.11.121.67) to abf53778adcf497787bbf7bac00509e2 (127.11.121.66). New cstate: current_term: 8 leader_uuid: "abf53778adcf497787bbf7bac00509e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:52:18.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:18.319818 13255 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 -> Peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Couldn't send request to peer 7c5d736667634e189b1c959e88b14aba. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:18.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:18.400177 13504 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 8 FOLLOWER]: Refusing update from remote peer abf53778adcf497787bbf7bac00509e2: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
05:52:18.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:18.401190 13563 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
05:52:20.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:20.951043 13255 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 -> Peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Couldn't send request to peer 7c5d736667634e189b1c959e88b14aba. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:52:21.423 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:21.424 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:21.425 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4846), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [3ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [24ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [26ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [28ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [28ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [41ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [44ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [46ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [48ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [84ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [85ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [87ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [87ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [222ms] refreshing cache from master, [222ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [224ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [226ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [228ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [228ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [261ms] refreshing cache from master, [262ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [264ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [265ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [267ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [268ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [802ms] refreshing cache from master, [802ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [804ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [808ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [810ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [810ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [4842ms] refreshing cache from master, [4842ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [4845ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@353492820(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@61dffc3a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117) -> (continuation of Deferred@1406928045 after retry RPC@1009215526) -> (continuation of Deferred@488642458 after retry RPC@440478272) -> (continuation of Deferred@1882188504 after retry RPC@1780337189), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1ff843ba -> wakeup thread Time-limited test -> (continuation of Deferred@1209682645 after retry RPC@362948095) -> (continuation of Deferred@1710607513 after retry RPC@1659622957) -> (continuation of Deferred@1734834618 after retry RPC@1535218117) -> (continuation of Deferred@1406928045 after retry RPC@1009215526) -> (continuation of Deferred@488642458 after retry RPC@440478272) -> (continuation of Deferred@1882188504 after retry RPC@1780337189))) after lookup
05:52:21.425 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:21.427 [DEBUG - kudu-nio-1] (Connection.java:757) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] Successfully connected to peer
05:52:21.446 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x1f124f0d, L:/127.0.0.1:58450 - R:/127.11.121.66:39283] using TOKEN/null
05:52:21.475 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.67:38133
05:52:21.476 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.476852 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:52:21.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:21.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:52:21.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:52:21.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:21.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:21.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:21.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:21.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.811471 13604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.811861 13604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.811996 13604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.822527 13604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:21.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.822762 13604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.67
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.831990 13604 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:52:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:21.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.833709 13604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:21.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.835532 13604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:21.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.849259 13612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:21.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.849248 13614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:21.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:21.849256 13611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:21.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.849877 13604 server_base.cc:1034] running on GCE node
05:52:21.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.851171 13604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:21.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.852324 13604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:21.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.853569 13604 hybrid_clock.cc:648] HybridClock initialized: now 1783489941853487 us; error 55 us; skew 500 ppm
05:52:21.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.853859 13604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:21.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.859901 13604 webserver.cc:460] Webserver started at http://127.11.121.67:40053/ using document root <none> and password file <none>
05:52:21.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.860733 13604 fs_manager.cc:359] Metadata directory not provided
05:52:21.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.860848 13604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:21.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.866787 13604 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
05:52:21.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.869882 13620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:21.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.871349 13604 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
05:52:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.871495 13604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7c5d736667634e189b1c959e88b14aba"
05:52:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:52:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.872612 13604 fs_report.cc:383] FS layout report
05:52:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/data
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:21.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.893307 13604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:21.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.894150 13604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:21.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.894415 13604 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:21.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.895426 13604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:21.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.897500 13627 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.903226 13604 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.903342 13604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
05:52:21.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.903532 13604 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:21.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.908058 13604 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:21.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.908141 13604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s	user 0.003s	sys 0.002s
05:52:21.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.908638 13627 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap starting.
05:52:21.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.935251 13627 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Log is configured to *not* fsync() on all Append() calls
05:52:21.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.958379 13604 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.67:38133
05:52:21.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.958398 13734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.67:38133 every 8 connection(s)
05:52:21.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.959667 13604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:21.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.964599 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 13604
05:52:21.965 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:21.984 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:21.984 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.66:39283
05:52:21.985 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:52:21.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:21.985325 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 13238
05:52:22.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.026800 13627 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:22.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.027508 13627 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap complete.
05:52:22.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.029074 13627 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent bootstrapping tablet: real 0.121s	user 0.038s	sys 0.027s
05:52:22.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.038957 13736 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:22.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.039234 13736 heartbeater.cc:464] Registering TS with master...
05:52:22.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.040339 13736 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:22.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.043439 11791 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:22.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.044982 13735 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:22.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.045106 13735 heartbeater.cc:464] Registering TS with master...
05:52:22.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.045586 13735 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:22.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.045042 13627 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:22.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.046613 13627 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Initialized, Role: FOLLOWER
05:52:22.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.047402 13627 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:22.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.050063 11932 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:22.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.050704 13737 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:22.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.050845 13737 heartbeater.cc:464] Registering TS with master...
05:52:22.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.051302 13737 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:22.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.052348 11932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.67:49915
05:52:22.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.052963 11862 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:22.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.060688 13735 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:22.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.061645 13627 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent starting tablet: real 0.032s	user 0.014s	sys 0.011s
05:52:22.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.069415 13362 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:22.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:22.192155 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 13238
05:52:22.215 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:22.215 [DEBUG - kudu-nio-1] (Connection.java:239) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] handling channelInactive
05:52:22.216 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state READY due to: connection closed
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:22.216459 13750 negotiation.cc:337] Failed RPC negotiation. Trace:
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.067360 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.121.66:43617 (local address 127.11.121.67:38133)
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.068463 (+  1103us) server_negotiation.cc:206] Beginning negotiation
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.068472 (+     9us) server_negotiation.cc:399] Waiting for connection header
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.069130 (+   658us) server_negotiation.cc:407] Connection header received
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.069237 (+   107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.069247 (+    10us) server_negotiation.cc:461] Received NEGOTIATE request from client
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.069566 (+   319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.069767 (+   201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
05:52:22.216 [DEBUG - Time-limited test] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.193013 (+123246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
05:52:22.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.194813 (+  1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
05:52:22.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 05:52:22.214768 (+ 19955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.121.66:43617: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
05:52:22.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":784,"threads_started":1}
05:52:22.218 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.218 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.218 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.222 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:22.222 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:22.222 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=6), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [2ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [6ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@28263446(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@6f59643d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cd90d7e -> wakeup thread Time-limited test)) after lookup
05:52:22.223 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:22.224 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.224 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.224 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.243 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:22.243 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:22.243 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [2ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [6ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [6ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [8ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [8ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [26ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@28263446(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@6f59643d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cd90d7e -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510))) after lookup
05:52:22.244 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:22.245 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.245 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.245 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.282 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:22.283 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:22.283 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [2ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [6ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [6ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [8ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [8ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [26ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [29ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [29ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [66ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@28263446(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@6f59643d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cd90d7e -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062))) after lookup
05:52:22.284 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:22.285 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.285 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.285 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.363 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:22.363 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:22.363 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=147), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [2ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [6ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [6ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [8ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [8ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [26ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [29ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [29ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [66ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [67ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [69ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [69ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [144ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [146ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@28263446(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@6f59643d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062) -> (continuation of Deferred@1082603082 after retry RPC@351320787), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cd90d7e -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062) -> (continuation of Deferred@1082603082 after retry RPC@351320787))) after lookup
05:52:22.364 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:22.365 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.365 [DEBUG - kudu-nio-0] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.365 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.722 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:22.723 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:22.723 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=507), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [2ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [6ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [6ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [8ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [8ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [26ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [29ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [29ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [66ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [67ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [69ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [69ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [144ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [146ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [148ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [149ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [149ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [504ms] refreshing cache from master, [504ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [506ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@28263446(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@6f59643d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062) -> (continuation of Deferred@1082603082 after retry RPC@351320787) -> (continuation of Deferred@941028755 after retry RPC@1927851682), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cd90d7e -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062) -> (continuation of Deferred@1082603082 after retry RPC@351320787) -> (continuation of Deferred@941028755 after retry RPC@1927851682))) after lookup
05:52:22.724 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:22.726 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.726 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state CONNECTING due to: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:22.726 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283
05:52:23.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.531180 13752 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader abf53778adcf497787bbf7bac00509e2)
05:52:23.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.531388 13752 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:23.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.532928 13752 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:23.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:23.534816 13435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:52:23.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:23.536331 13435 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:52:23.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.541009 13689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba" is_pre_election: true
05:52:23.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.541456 13689 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2f80c10be94511bdabb93db3d6d160 in term 7.
05:52:23.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.542456 13437 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160; no voters: abf53778adcf497787bbf7bac00509e2
05:52:23.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.542933 13752 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 8 FOLLOWER]: Leader pre-election won for term 9
05:52:23.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.543071 13752 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 8 FOLLOWER]: Starting leader election (detected failure of leader abf53778adcf497787bbf7bac00509e2)
05:52:23.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.543135 13752 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 8 FOLLOWER]: Advancing to term 9
05:52:23.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.545104 13752 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:23.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.545678 13752 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 9 election: Requested vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:23.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.546322 13689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba"
05:52:23.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.546581 13689 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 7 FOLLOWER]: Advancing to term 9
05:52:23.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:23.547308 13435 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111)
05:52:23.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.551009 13689 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2f80c10be94511bdabb93db3d6d160 in term 9.
05:52:23.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.551564 13437 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160; no voters: abf53778adcf497787bbf7bac00509e2
05:52:23.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.551894 13752 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Leader election won for term 9
05:52:23.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.552412 13752 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 LEADER]: Becoming Leader. State: Replica: 8d2f80c10be94511bdabb93db3d6d160, State: Running, Role: LEADER
05:52:23.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.552897 13752 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:23.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:23.556427 11932 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 reported cstate change: term changed from 8 to 9, leader changed from abf53778adcf497787bbf7bac00509e2 (127.11.121.66) to 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65). New cstate: current_term: 9 leader_uuid: "8d2f80c10be94511bdabb93db3d6d160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: HEALTHY } } }
05:52:24.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.012820 13689 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 9 FOLLOWER]: Refusing update from remote peer 8d2f80c10be94511bdabb93db3d6d160: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
05:52:24.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.014088 13752 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
05:52:24.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.027292 13435 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 -> Peer abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283): Couldn't send request to peer abf53778adcf497787bbf7bac00509e2. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.66:39283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:24.182 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:24.183 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:24.183 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1967), Traces: [0ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [2ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [2ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [3ms] refreshing cache from master, [3ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [6ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [6ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [8ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [8ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [26ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [28ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [29ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [29ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [66ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [67ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [69ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [69ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [144ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [146ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [148ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [149ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [149ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [504ms] refreshing cache from master, [504ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [506ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [507ms] sending RPC to server abf53778adcf497787bbf7bac00509e2, [510ms] received response from server abf53778adcf497787bbf7bac00509e2: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [510ms] delaying RPC due to: Network error: Failed to connect to peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283): Connection refused: /127.11.121.66:39283, [1964ms] refreshing cache from master, [1964ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [1966ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@28263446(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@6f59643d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062) -> (continuation of Deferred@1082603082 after retry RPC@351320787) -> (continuation of Deferred@941028755 after retry RPC@1927851682) -> (continuation of Deferred@1413627473 after retry RPC@1667762117), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3cd90d7e -> wakeup thread Time-limited test -> (continuation of Deferred@1898174137 after retry RPC@98932510) -> (continuation of Deferred@418010123 after retry RPC@43702062) -> (continuation of Deferred@1082603082 after retry RPC@351320787) -> (continuation of Deferred@941028755 after retry RPC@1927851682) -> (continuation of Deferred@1413627473 after retry RPC@1667762117))) after lookup
05:52:24.184 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:24.185 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] Successfully connected to peer
05:52:24.201 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x68dbf00d, L:/127.0.0.1:60816 - R:/127.11.121.65:46477] using TOKEN/null
05:52:24.223 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.66:39283
05:52:24.224 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

05:52:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.224706 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:52:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:39283
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43205
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:24.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.585619 13779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:24.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.585911 13779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:24.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.585994 13779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:24.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.594252 13779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:24.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.594460 13779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.66
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.604470 13779 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.66:39283
05:52:24.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.66
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43205
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.66
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/logs
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:24.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.605866 13779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:24.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.607640 13779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.621517 13786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.621475 13789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.621564 13787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:24.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.622965 13779 server_base.cc:1034] running on GCE node
05:52:24.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.623588 13779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:24.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.624632 13779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:24.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.625888 13779 hybrid_clock.cc:648] HybridClock initialized: now 1783489944625795 us; error 64 us; skew 500 ppm
05:52:24.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.626241 13779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:24.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.628757 13779 webserver.cc:460] Webserver started at http://127.11.121.66:43205/ using document root <none> and password file <none>
05:52:24.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.629487 13779 fs_manager.cc:359] Metadata directory not provided
05:52:24.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.629632 13779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:24.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.635095 13779 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
05:52:24.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.638495 13795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:24.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.640017 13779 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.640242 13779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "abf53778adcf497787bbf7bac00509e2"
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.641400 13779 fs_report.cc:383] FS layout report
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/wal
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/data
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:24.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.659276 13779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:24.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.660280 13779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:24.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.660625 13779 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:24.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.661687 13779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:24.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.663694 13802 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:24.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.669251 13779 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:24.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.669396 13779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
05:52:24.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.669535 13779 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:24.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.672801 13779 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:24.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.672909 13779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
05:52:24.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.673245 13802 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap starting.
05:52:24.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.699862 13802 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Log is configured to *not* fsync() on all Append() calls
05:52:24.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.725972 13779 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.66:39283
05:52:24.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.725992 13909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.66:39283 every 8 connection(s)
05:52:24.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.727267 13779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-1/data/info.pb
05:52:24.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.732719 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 13779
05:52:24.733 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:24.746 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:24.747 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.65:46477
05:52:24.747 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:52:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.748086 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 13420
05:52:24.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.774022 13910 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:24.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.774390 13910 heartbeater.cc:464] Registering TS with master...
05:52:24.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.775418 13910 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:24.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.778294 11931 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:52:24.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.779062 13802 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:24.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.779731 13802 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Bootstrap complete.
05:52:24.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.780577 11931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.66:39375
05:52:24.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.781375 13802 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent bootstrapping tablet: real 0.108s	user 0.060s	sys 0.013s
05:52:24.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.784088 13544 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:24.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.784207 13911 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:24.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.784345 13911 heartbeater.cc:464] Registering TS with master...
05:52:24.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.785064 13911 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:24.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.785205 13912 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:24.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.785341 13912 heartbeater.cc:464] Registering TS with master...
05:52:24.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.785755 13912 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:24.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.786702 11791 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:52:24.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.787197 11862 ts_manager.cc:194] Re-registered known tserver with Master: abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283)
05:52:24.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.792331 13802 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:24.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.793232 13802 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Initialized, Role: FOLLOWER
05:52:24.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.793990 13802 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:24.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.795143 13910 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:24.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.796008 13802 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.004s
05:52:24.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:24.884121 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 13420
05:52:24.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:24.902223 13623 connection.cc:535] server connection from 127.11.121.65:52355 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
05:52:24.903 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:24.903 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] handling channelInactive
05:52:24.903 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state READY due to: connection closed
05:52:24.903 [DEBUG - Time-limited test] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:24.905 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.905 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.905 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.923 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:24.923 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:24.923 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [19ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@904521293(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@74647bd3 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d81cc1a -> wakeup thread Time-limited test)) after lookup
05:52:24.924 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:24.924 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.924 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.925 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.942 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:24.942 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:24.943 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [19ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [21ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [22ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [39ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@904521293(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@74647bd3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d81cc1a -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338))) after lookup
05:52:24.943 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:24.944 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.944 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.945 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.963 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:24.964 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:24.964 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=60), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [19ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [21ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [22ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [39ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [40ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [41ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [42ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [60ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@904521293(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@74647bd3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d81cc1a -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261))) after lookup
05:52:24.964 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:24.965 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.965 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:24.966 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.003 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:25.003 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:25.004 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=100), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [19ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [21ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [22ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [39ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [40ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [41ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [42ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [60ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [61ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [62ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [63ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [97ms] refreshing cache from master, [97ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [99ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@904521293(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@74647bd3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d81cc1a -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420))) after lookup
05:52:25.004 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:25.005 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.005 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.005 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.243 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:25.243 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:25.243 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=340), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [19ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [21ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [22ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [39ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [40ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [41ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [42ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [60ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [61ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [62ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [63ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [97ms] refreshing cache from master, [97ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [99ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [101ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [102ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [102ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [337ms] refreshing cache from master, [337ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [339ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@904521293(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@74647bd3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420) -> (continuation of Deferred@502480767 after retry RPC@1759206480), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d81cc1a -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420) -> (continuation of Deferred@502480767 after retry RPC@1759206480))) after lookup
05:52:25.244 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:25.244 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.245 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.245 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.603 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:25.604 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)[L],abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:25.604 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=701), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [19ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [21ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [22ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [39ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [40ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [41ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [42ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [60ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [61ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [62ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [63ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [97ms] refreshing cache from master, [97ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [99ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [101ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [102ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [102ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [337ms] refreshing cache from master, [337ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [339ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [340ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [342ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [342ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [697ms] refreshing cache from master, [697ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [700ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@904521293(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@74647bd3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420) -> (continuation of Deferred@502480767 after retry RPC@1759206480) -> (continuation of Deferred@1269191084 after retry RPC@1338960957), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d81cc1a -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420) -> (continuation of Deferred@502480767 after retry RPC@1759206480) -> (continuation of Deferred@1269191084 after retry RPC@1338960957))) after lookup
05:52:25.605 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] connecting to peer
05:52:25.606 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.606 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] cleaning up while in state CONNECTING due to: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:25.606 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477
05:52:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.320294 13925 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:52:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.320604 13925 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:26.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.322386 13925 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:26.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:26.324568 13799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:26.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.329833 13689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "7c5d736667634e189b1c959e88b14aba" is_pre_election: true
05:52:26.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:26.330592 13799 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:26.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.330844 13799 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba, 8d2f80c10be94511bdabb93db3d6d160
05:52:26.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.331235 13925 raft_consensus.cc:2726] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
05:52:26.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.385540 13930 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8d2f80c10be94511bdabb93db3d6d160)
05:52:26.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.385844 13930 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:26.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.387567 13930 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:26.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:26.389143 13624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:26.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:26.390944 13624 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:26.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.397414 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2" is_pre_election: true
05:52:26.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.397889 13865 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 8.
05:52:26.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.398929 13621 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2; no voters: 8d2f80c10be94511bdabb93db3d6d160
05:52:26.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.399358 13930 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 9 FOLLOWER]: Leader pre-election won for term 10
05:52:26.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.399504 13930 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8d2f80c10be94511bdabb93db3d6d160)
05:52:26.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.399587 13930 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 9 FOLLOWER]: Advancing to term 10
05:52:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.401942 13930 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.402629 13930 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 10 election: Requested vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:26.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.403026 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "7c5d736667634e189b1c959e88b14aba" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2"
05:52:26.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.403244 13865 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 8 FOLLOWER]: Advancing to term 10
05:52:26.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:26.404413 13624 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111)
05:52:26.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.407799 13865 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c5d736667634e189b1c959e88b14aba in term 10.
05:52:26.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.408412 13621 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2; no voters: 8d2f80c10be94511bdabb93db3d6d160
05:52:26.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.408722 13930 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 10 FOLLOWER]: Leader election won for term 10
05:52:26.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.409173 13930 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 10 LEADER]: Becoming Leader. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Running, Role: LEADER
05:52:26.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.409708 13930 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:26.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.413461 11931 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba reported cstate change: term changed from 9 to 10, leader changed from 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65) to 7c5d736667634e189b1c959e88b14aba (127.11.121.67). New cstate: current_term: 10 leader_uuid: "7c5d736667634e189b1c959e88b14aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:52:26.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.850437 13865 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 10 FOLLOWER]: Refusing update from remote peer 7c5d736667634e189b1c959e88b14aba: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
05:52:26.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:26.851588 13930 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
05:52:26.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:26.863257 13624 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba -> Peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Couldn't send request to peer 8d2f80c10be94511bdabb93db3d6d160. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:29.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.171487 13624 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba -> Peer 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477): Couldn't send request to peer 8d2f80c10be94511bdabb93db3d6d160. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.65:46477: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
05:52:29.323 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:29.323 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:29.323 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=4420), Traces: [0ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [2ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [3ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [19ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [20ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [21ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [22ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [39ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [40ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [41ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [42ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [60ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [61ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [62ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [63ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [97ms] refreshing cache from master, [97ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [99ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [101ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [102ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [102ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [337ms] refreshing cache from master, [337ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [339ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [340ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [342ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [342ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [697ms] refreshing cache from master, [697ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [700ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [702ms] sending RPC to server 8d2f80c10be94511bdabb93db3d6d160, [703ms] received response from server 8d2f80c10be94511bdabb93db3d6d160: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [703ms] delaying RPC due to: Network error: Failed to connect to peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477): Connection refused: /127.11.121.65:46477, [4416ms] refreshing cache from master, [4417ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [4419ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@904521293(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@74647bd3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420) -> (continuation of Deferred@502480767 after retry RPC@1759206480) -> (continuation of Deferred@1269191084 after retry RPC@1338960957) -> (continuation of Deferred@1528790115 after retry RPC@1669909561), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3d81cc1a -> wakeup thread Time-limited test -> (continuation of Deferred@1115132429 after retry RPC@1795041338) -> (continuation of Deferred@818951453 after retry RPC@729048261) -> (continuation of Deferred@688388202 after retry RPC@1395973420) -> (continuation of Deferred@502480767 after retry RPC@1759206480) -> (continuation of Deferred@1269191084 after retry RPC@1338960957) -> (continuation of Deferred@1528790115 after retry RPC@1669909561))) after lookup
05:52:29.324 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:29.325 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] Successfully connected to peer
05:52:29.346 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x9e513b84, L:/127.0.0.1:44802 - R:/127.11.121.67:38133] using TOKEN/null
05:52:29.369 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.65:46477
05:52:29.369 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.370365 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:46477
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34993
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:29.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.699556 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:29.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.699931 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:29.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.700062 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:29.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.708190 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:29.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.708422 13968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.65
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.718043 13968 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.65:46477
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.65
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34993
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.65
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/logs
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:29.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.719709 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.721724 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:29.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.736502 13974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:29.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.736532 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:29.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:29.737047 13973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:29.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.737709 13968 server_base.cc:1034] running on GCE node
05:52:29.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.738327 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:29.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.739306 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:29.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.740532 13968 hybrid_clock.cc:648] HybridClock initialized: now 1783489949740437 us; error 61 us; skew 500 ppm
05:52:29.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.740901 13968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:29.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.747717 13968 webserver.cc:460] Webserver started at http://127.11.121.65:34993/ using document root <none> and password file <none>
05:52:29.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.748489 13968 fs_manager.cc:359] Metadata directory not provided
05:52:29.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.748653 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:29.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.753832 13968 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
05:52:29.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.756915 13982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:29.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.758301 13968 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
05:52:29.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.758486 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:29.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:51 on dist-test-slave-w6r9"
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.759498 13968 fs_report.cc:383] FS layout report
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/wal
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/data
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:29.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:29.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:29.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.794606 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:29.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.795557 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:29.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.795886 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:29.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.797011 13968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:29.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.798967 13989 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:29.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.804618 13968 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:29.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.804756 13968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
05:52:29.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.804889 13968 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:29.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.808156 13968 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:29.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.808267 13968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
05:52:29.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.808689 13989 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap starting.
05:52:29.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.837922 13989 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Log is configured to *not* fsync() on all Append() calls
05:52:29.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.851612 13968 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.65:46477
05:52:29.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.851706 14096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.65:46477 every 8 connection(s)
05:52:29.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.853056 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-0/data/info.pb
05:52:29.858 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:29.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.858004 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 13968
05:52:29.904 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:29.905 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.11.121.67:38133
05:52:29.905 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:52:29.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.906311 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 13604
05:52:29.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.922045 13989 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:29.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.922914 13989 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Bootstrap complete.
05:52:29.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.924664 13989 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent bootstrapping tablet: real 0.116s	user 0.061s	sys 0.020s
05:52:29.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.927436 14097 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:29.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.927776 14097 heartbeater.cc:464] Registering TS with master...
05:52:29.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.928923 14097 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:29.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.932089 11932 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:29.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.933697 11932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.65:57505
05:52:29.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.933605 13989 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:29.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.934917 13989 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2f80c10be94511bdabb93db3d6d160, State: Initialized, Role: FOLLOWER
05:52:29.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.935658 13989 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:29.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.935986 13730 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.938490 14098 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.938642 14098 heartbeater.cc:464] Registering TS with master...
05:52:29.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.939049 14098 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:29.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.940403 11790 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:29.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.945626 14099 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:29.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.945776 14099 heartbeater.cc:464] Registering TS with master...
05:52:29.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.946230 14099 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:29.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.947032 14097 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:29.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.947036 13989 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160: Time spent starting tablet: real 0.022s	user 0.016s	sys 0.005s
05:52:29.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:29.948074 11862 ts_manager.cc:194] Re-registered known tserver with Master: 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:30.044082 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 13604
05:52:30.062 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:30.062 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] handling channelInactive
05:52:30.062 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state READY due to: connection closed
05:52:30.063 [DEBUG - Time-limited test] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:30.064 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.064 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.064 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.082 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:30.083 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:30.083 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@122681004(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@17a1b6f2 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@45cff724 -> wakeup thread Time-limited test)) after lookup
05:52:30.083 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:30.084 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.084 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.084 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.102 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:30.103 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:30.103 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [21ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [22ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [23ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@122681004(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@17a1b6f2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@45cff724 -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270))) after lookup
05:52:30.104 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:30.104 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.105 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.105 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.142 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:30.143 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:30.143 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [21ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [22ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [23ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [41ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [43ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [80ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@122681004(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@17a1b6f2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@45cff724 -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287))) after lookup
05:52:30.144 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:30.145 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.145 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.145 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.282 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:30.282 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:30.283 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=220), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [21ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [22ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [23ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [41ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [43ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [80ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [81ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [83ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [83ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [217ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [220ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@122681004(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@17a1b6f2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287) -> (continuation of Deferred@1183736690 after retry RPC@1994995271), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@45cff724 -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287) -> (continuation of Deferred@1183736690 after retry RPC@1994995271))) after lookup
05:52:30.284 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:30.285 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.285 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.285 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.543 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:30.543 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)[L],8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)]}]
05:52:30.543 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=480), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [21ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [22ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [23ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [41ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [43ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [80ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [81ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [83ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [83ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [217ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [220ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [221ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [223ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [223ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [478ms] refreshing cache from master, [478ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [480ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@122681004(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@17a1b6f2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287) -> (continuation of Deferred@1183736690 after retry RPC@1994995271) -> (continuation of Deferred@1476400822 after retry RPC@1743018239), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@45cff724 -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287) -> (continuation of Deferred@1183736690 after retry RPC@1994995271) -> (continuation of Deferred@1476400822 after retry RPC@1743018239))) after lookup
05:52:30.544 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] connecting to peer
05:52:30.545 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.545 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:30.545 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133) for tablet 2ab79f7ab2774f3ead34bc6df3d5c50f: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133
05:52:31.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.232616 14112 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
05:52:31.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.232860 14112 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:31.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.234548 14112 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers abf53778adcf497787bbf7bac00509e2 (127.11.121.66:39283), 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:31.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:31.236310 13985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:31.237888 13985 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:31.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.241930 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "8d2f80c10be94511bdabb93db3d6d160" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "abf53778adcf497787bbf7bac00509e2" is_pre_election: true
05:52:31.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.242833 13983 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160; no voters: 7c5d736667634e189b1c959e88b14aba, abf53778adcf497787bbf7bac00509e2
05:52:31.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.243186 14112 raft_consensus.cc:2726] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
05:52:31.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.356082 14117 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:52:31.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.356266 14117 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:31.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.356935 14117 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:31.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:31.358803 13798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:31.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:31.360409 13798 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:31.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.365304 14051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160" is_pre_election: true
05:52:31.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.365698 14051 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 9.
05:52:31.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.366693 13799 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:52:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.367060 14117 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 10 FOLLOWER]: Leader pre-election won for term 11
05:52:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.367188 14117 raft_consensus.cc:482] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7c5d736667634e189b1c959e88b14aba)
05:52:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.367295 14117 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 10 FOLLOWER]: Advancing to term 11
05:52:31.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.369370 14117 raft_consensus.cc:504] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:31.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.369979 14117 leader_election.cc:288] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 11 election: Requested vote from peers 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133), 8d2f80c10be94511bdabb93db3d6d160 (127.11.121.65:46477)
05:52:31.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.370487 14051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab79f7ab2774f3ead34bc6df3d5c50f" candidate_uuid: "abf53778adcf497787bbf7bac00509e2" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "8d2f80c10be94511bdabb93db3d6d160"
05:52:31.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.370692 14051 raft_consensus.cc:3037] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 9 FOLLOWER]: Advancing to term 11
05:52:31.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:31.371625 13798 leader_election.cc:334] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111)
05:52:31.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.374933 14051 raft_consensus.cc:2454] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abf53778adcf497787bbf7bac00509e2 in term 11.
05:52:31.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.375375 13799 leader_election.cc:302] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d2f80c10be94511bdabb93db3d6d160, abf53778adcf497787bbf7bac00509e2; no voters: 7c5d736667634e189b1c959e88b14aba
05:52:31.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.375715 14117 raft_consensus.cc:2781] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 11 FOLLOWER]: Leader election won for term 11
05:52:31.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.376240 14117 raft_consensus.cc:686] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [term 11 LEADER]: Becoming Leader. State: Replica: abf53778adcf497787bbf7bac00509e2, State: Running, Role: LEADER
05:52:31.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.376761 14117 consensus_queue.cc:237] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:31.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.380615 11932 catalog_manager.cc:5495] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 reported cstate change: term changed from 10 to 11, leader changed from 7c5d736667634e189b1c959e88b14aba (127.11.121.67) to abf53778adcf497787bbf7bac00509e2 (127.11.121.66). New cstate: current_term: 11 leader_uuid: "abf53778adcf497787bbf7bac00509e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } health_report { overall_health: UNKNOWN } } }
05:52:31.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.861316 14051 raft_consensus.cc:1261] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 8d2f80c10be94511bdabb93db3d6d160 [term 11 FOLLOWER]: Refusing update from remote peer abf53778adcf497787bbf7bac00509e2: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
05:52:31.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:31.862179 14117 consensus_queue.cc:1035] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
05:52:31.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:31.893103 13798 consensus_peers.cc:487] T 2ab79f7ab2774f3ead34bc6df3d5c50f P abf53778adcf497787bbf7bac00509e2 -> Peer 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133): Couldn't send request to peer 7c5d736667634e189b1c959e88b14aba. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.67:38133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:32.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet 2ab79f7ab2774f3ead34bc6df3d5c50f for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204' with partition [<start>, <end>)
05:52:32.522 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f@[7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133),8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477),abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)[L]]}]
05:52:32.523 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2460), Traces: [0ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [2ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [20ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [21ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [22ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [23ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [40ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [41ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [43ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [43ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [80ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [81ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [83ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [83ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [217ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [220ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [221ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [223ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [223ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [478ms] refreshing cache from master, [478ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [480ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, [482ms] sending RPC to server 7c5d736667634e189b1c959e88b14aba, [483ms] received response from server 7c5d736667634e189b1c959e88b14aba: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [483ms] delaying RPC due to: Network error: Failed to connect to peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133): Connection refused: /127.11.121.67:38133, [2457ms] refreshing cache from master, [2457ms] Sub RPC GetTableLocations: sending RPC to server master-127.11.121.124:33337, [2459ms] Sub RPC GetTableLocations: received response from server master-127.11.121.124:33337: OK, deferred=Deferred@122681004(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@17a1b6f2 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287) -> (continuation of Deferred@1183736690 after retry RPC@1994995271) -> (continuation of Deferred@1476400822 after retry RPC@1743018239) -> (continuation of Deferred@1301012758 after retry RPC@787403895), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@45cff724 -> wakeup thread Time-limited test -> (continuation of Deferred@397823975 after retry RPC@1401100270) -> (continuation of Deferred@703719336 after retry RPC@1652992287) -> (continuation of Deferred@1183736690 after retry RPC@1994995271) -> (continuation of Deferred@1476400822 after retry RPC@1743018239) -> (continuation of Deferred@1301012758 after retry RPC@787403895))) after lookup
05:52:32.523 [DEBUG - kudu-nio-0] (Connection.java:748) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] connecting to peer
05:52:32.524 [DEBUG - kudu-nio-1] (Connection.java:757) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] Successfully connected to peer
05:52:32.542 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x389ce38b, L:/127.0.0.1:58646 - R:/127.11.121.66:39283] using TOKEN/null
05:52:32.564 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.11.121.67:38133
05:52:32.564 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.565245 11749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:32.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
05:52:32.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.908443 14154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
05:52:32.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.908795 14154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
05:52:32.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.908891 14154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
05:52:32.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.916599 14154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
05:52:32.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.916798 14154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.121.67
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.926379 14154 tablet_server_runner.cc:78] Tablet server non-default flags:
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.11.121.84:46187
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.11.121.67:38133
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.11.121.67
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40053
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.11.121.126:37585,127.11.121.125:38073,127.11.121.124:33337
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.11.121.67
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/logs
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Jul 2026 03:56:09 UTC on ccycloud.quasar-kartrx.root.comops.site
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4492622
05:52:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
05:52:32.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.927763 14154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
05:52:32.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.929615 14154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
05:52:32.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.943994 14159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:32.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.944070 14160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:32.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:32.944475 14162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
05:52:32.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.945315 14154 server_base.cc:1034] running on GCE node
05:52:32.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.945992 14154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
05:52:32.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.947012 14154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
05:52:32.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.948249 14154 hybrid_clock.cc:648] HybridClock initialized: now 1783489952948169 us; error 46 us; skew 500 ppm
05:52:32.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.948552 14154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
05:52:32.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.951344 14154 webserver.cc:460] Webserver started at http://127.11.121.67:40053/ using document root <none> and password file <none>
05:52:32.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.952497 14154 fs_manager.cc:359] Metadata directory not provided
05:52:32.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.952647 14154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
05:52:32.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.960239 14154 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.002s
05:52:32.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.963812 14168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
05:52:32.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.965775 14154 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.965981 14154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data,/tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7c5d736667634e189b1c959e88b14aba"
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 05:51:52 on dist-test-slave-w6r9"
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.967500 14154 fs_report.cc:383] FS layout report
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/wal
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/data
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
05:52:32.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
05:52:32.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
05:52:32.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.982107 14154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
05:52:32.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.982896 14154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
05:52:32.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.983131 14154 kserver.cc:163] Server-wide thread pool size limit: 3276
05:52:32.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.984143 14154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
05:52:32.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.986059 14175 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
05:52:32.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.992240 14154 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
05:52:32.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.992367 14154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
05:52:32.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.992465 14154 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
05:52:32.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.995669 14154 ts_tablet_manager.cc:610] Registered 1 tablets
05:52:32.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.995755 14154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
05:52:32.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:32.996253 14175 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap starting.
05:52:33.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.026324 14175 log.cc:824] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Log is configured to *not* fsync() on all Append() calls
05:52:33.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.049106 14154 rpc_server.cc:307] RPC server started. Bound to: 127.11.121.67:38133
05:52:33.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.049145 14284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.121.67:38133 every 8 connection(s)
05:52:33.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.050462 14154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGJ_3KX/test-tmp/mini-kudu-cluster3112301352984600499/ts-2/data/info.pb
05:52:33.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.052825 11749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu as pid 14154
05:52:33.053 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
05:52:33.084 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:33.092 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783489913204, tablet=2ab79f7ab2774f3ead34bc6df3d5c50f, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
05:52:33.098 [DEBUG - main] (Connection.java:575) [peer 8d2f80c10be94511bdabb93db3d6d160(127.11.121.65:46477)] disconnecting while in state TERMINATED
05:52:33.101 [DEBUG - main] (Connection.java:575) [peer master-127.11.121.126:37585(127.11.121.126:37585)] disconnecting while in state READY
05:52:33.104 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.11.121.126:37585(127.11.121.126:37585)] handling channelInactive
05:52:33.104 [DEBUG - main] (Connection.java:575) [peer 7c5d736667634e189b1c959e88b14aba(127.11.121.67:38133)] disconnecting while in state TERMINATED
05:52:33.104 [DEBUG - main] (Connection.java:575) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] disconnecting while in state READY
05:52:33.104 [DEBUG - main] (Connection.java:575) [peer master-127.11.121.125:38073(127.11.121.125:38073)] disconnecting while in state READY
05:52:33.105 [DEBUG - main] (Connection.java:575) [peer master-127.11.121.124:33337(127.11.121.124:33337)] disconnecting while in state READY
05:52:33.105 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.11.121.126:37585(127.11.121.126:37585)] cleaning up while in state READY due to: connection closed
05:52:33.106 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.11.121.125:38073(127.11.121.125:38073)] handling channelInactive
05:52:33.107 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.11.121.125:38073(127.11.121.125:38073)] cleaning up while in state READY due to: connection closed
05:52:33.107 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.11.121.124:33337(127.11.121.124:33337)] handling channelInactive
05:52:33.107 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.11.121.124:33337(127.11.121.124:33337)] cleaning up while in state READY due to: connection closed
05:52:33.107 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
05:52:33.108 [DEBUG - kudu-nio-1] (Connection.java:239) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] handling channelInactive
05:52:33.109 [DEBUG - kudu-nio-1] (Connection.java:709) [peer abf53778adcf497787bbf7bac00509e2(127.11.121.66:39283)] cleaning up while in state READY due to: connection closed
05:52:33.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.109783 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 13968
05:52:33.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.126688 14285 heartbeater.cc:347] Connected to a master server at 127.11.121.124:33337
05:52:33.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.127120 14285 heartbeater.cc:464] Registering TS with master...
05:52:33.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.128255 14285 heartbeater.cc:511] Master 127.11.121.124:33337 requested a full tablet report, sending...
05:52:33.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.131644 11932 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:33.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.132351 14287 heartbeater.cc:347] Connected to a master server at 127.11.121.125:38073
05:52:33.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.132495 14287 heartbeater.cc:464] Registering TS with master...
05:52:33.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.132771 14286 heartbeater.cc:347] Connected to a master server at 127.11.121.126:37585
05:52:33.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.132949 14286 heartbeater.cc:464] Registering TS with master...
05:52:33.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.133008 14287 heartbeater.cc:511] Master 127.11.121.125:38073 requested a full tablet report, sending...
05:52:33.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.133239 11932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.121.67:58553
05:52:33.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.133466 14286 heartbeater.cc:511] Master 127.11.121.126:37585 requested a full tablet report, sending...
05:52:33.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.135208 11790 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:33.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.135356 11862 ts_manager.cc:194] Re-registered known tserver with Master: 7c5d736667634e189b1c959e88b14aba (127.11.121.67:38133)
05:52:33.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.135833 14175 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
05:52:33.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.136593 14175 tablet_bootstrap.cc:492] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Bootstrap complete.
05:52:33.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.138034 14175 ts_tablet_manager.cc:1397] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent bootstrapping tablet: real 0.142s	user 0.061s	sys 0.019s
05:52:33.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.144804 14175 raft_consensus.cc:348] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:33.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.146152 14175 raft_consensus.cc:729] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5d736667634e189b1c959e88b14aba, State: Initialized, Role: FOLLOWER
05:52:33.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.147253 14175 consensus_queue.cc:260] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf53778adcf497787bbf7bac00509e2" member_type: VOTER last_known_addr { host: "127.11.121.66" port: 39283 } } peers { permanent_uuid: "7c5d736667634e189b1c959e88b14aba" member_type: VOTER last_known_addr { host: "127.11.121.67" port: 38133 } } peers { permanent_uuid: "8d2f80c10be94511bdabb93db3d6d160" member_type: VOTER last_known_addr { host: "127.11.121.65" port: 46477 } }
05:52:33.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.149654 14092 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:33.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.150199 14285 heartbeater.cc:503] Master 127.11.121.124:33337 was elected leader, sending a full tablet report...
05:52:33.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.150903 14175 ts_tablet_manager.cc:1428] T 2ab79f7ab2774f3ead34bc6df3d5c50f P 7c5d736667634e189b1c959e88b14aba: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.005s
05:52:33.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.257082 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 13968
05:52:33.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.280364 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 13779
05:52:33.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.288724 13905 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:33.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.403522 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 13779
05:52:33.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.423030 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 14154
05:52:33.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.432245 14280 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:33.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.521457 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 14154
05:52:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.538328 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 11760
05:52:33.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.539538 11821 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:33.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.654862 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 11760
05:52:33.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.676962 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 11828
05:52:33.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.678135 11892 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:33.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:33.698719 11919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.121.126:37585: connect: Connection refused (error 111)
05:52:33.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 05:52:33.700825 11919 consensus_peers.cc:487] T 00000000000000000000000000000000 P 84db27231add4d86ae0c9105b822b26f -> Peer 3a64352b29874c3f8f14d5019b205f16 (127.11.121.126:37585): Couldn't send request to peer 3a64352b29874c3f8f14d5019b205f16. Status: Network error: Client connection negotiation failed: client connection to 127.11.121.126:37585: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
05:52:33.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.798115 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 11828
05:52:33.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.819766 11749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu pid 11899
05:52:33.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.820952 11963 generic_service.cc:196] Checking for leaks (request via RPC)
05:52:33.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 05:52:33.938768 11749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGJ_3KX/build/asan/bin/kudu with pid 11899
05:52:33.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T05:52:33Z chronyd exiting

Time: 46.058

OK (1 test)

