JUnit version 4.13.2
.18:50:07.117 [INFO - main] (RandomUtils.java:49) Using random seed: 1786474207110
18:50:07.164 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
18:50:07.226 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
18:50:07.239 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task6_Crsr/test-tmp
18:50:07.242 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
18:50:07.247 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task6_Crsr/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
18:50:07.469 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

18:50:07.669 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
18:50:07.670 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

18:50:07.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:07.674842  3461 mini_oidc.cc:152] Starting JWKS server
18:50:07.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:07.679752  3461 webserver.cc:460] Webserver started at http://127.0.0.1:43897/ using document root <none> and password file <none>
18:50:07.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:07.680086  3461 mini_oidc.cc:189] Starting OIDC Discovery server
18:50:07.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:07.680914  3461 webserver.cc:460] Webserver started at http://127.0.0.1:44579/ using document root <none> and password file <none>
18:50:07.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T18:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
18:50:07.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T18:50:07Z Disabled control of system clock
18:50:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:07.721529  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/wal
18:50:07.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data
18:50:07.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:07.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:07.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:07.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:07.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:07.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:07.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/logs
18:50:07.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data/info.pb
18:50:07.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:07.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:07.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:07.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:07.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:07.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:07.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:07.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:07.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:50:07.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:07.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:50:07.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:50:07.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.126:46829
18:50:07.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.126
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:07.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:50:07.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43897/jwks.json with env {}
18:50:08.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.078861  3475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:08.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.079208  3475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:08.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.079303  3475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:08.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.087339  3475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
18:50:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.087471  3475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.087515  3475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
18:50:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.087567  3475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
18:50:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.087606  3475 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
18:50:08.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.087718  3475 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:43897/jwks.json
18:50:08.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.103489  3475 master_runner.cc:386] Master server non-default flags:
18:50:08.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:08.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:08.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:08.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:08.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data
18:50:08.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/wal
18:50:08.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.126:46829
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43897/jwks.json
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:08.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data/info.pb
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.126
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/logs
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:08.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.105473  3475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:08.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.107861  3475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:08.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.121411  3480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:08.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.121464  3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:08.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.121466  3481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:08.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.121872  3475 server_base.cc:1034] running on GCE node
18:50:08.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.123370  3475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:08.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.125082  3475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:08.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.126364  3475 hybrid_clock.cc:648] HybridClock initialized: now 1786474208126273 us; error 70 us; skew 500 ppm
18:50:08.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.126885  3475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:08.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.133965  3475 webserver.cc:460] Webserver started at http://127.3.97.126:33553/ using document root <none> and password file <none>
18:50:08.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.134821  3475 fs_manager.cc:359] Metadata directory not provided
18:50:08.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.134950  3475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:08.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.135330  3475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
18:50:08.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.138273  3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data/instance:
18:50:08.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6a547507249f4364acc4c841f513a707"
18:50:08.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:08 on dist-test-slave-pgkr"
18:50:08.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.139207  3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/wal/instance:
18:50:08.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6a547507249f4364acc4c841f513a707"
18:50:08.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:08 on dist-test-slave-pgkr"
18:50:08.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.144935  3475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
18:50:08.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.148949  3489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:08.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.150516  3475 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
18:50:08.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.150765  3475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/wal
18:50:08.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6a547507249f4364acc4c841f513a707"
18:50:08.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:08 on dist-test-slave-pgkr"
18:50:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.151016  3475 fs_report.cc:383] FS layout report
18:50:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/wal
18:50:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/wal
18:50:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data/data
18:50:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:08.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:08.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.174808  3475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:08.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.175690  3475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:08.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.175967  3475 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:08.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.195947  3475 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.126:46829
18:50:08.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.195992  3550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.126:46829 every 8 connection(s)
18:50:08.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.197365  3475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/data/info.pb
18:50:08.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.201309  3551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:50:08.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.205289  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 3475
18:50:08.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.205590  3461 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-0/wal/instance
18:50:08.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.208597  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:08.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/wal
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/logs
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data/info.pb
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:08.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.125:45183
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.125
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:08.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:50:08.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43897/jwks.json with env {}
18:50:08.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.209613  3551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } has no permanent_uuid. Determining permanent_uuid...
18:50:08.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.229131  3551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } has no permanent_uuid. Determining permanent_uuid...
18:50:08.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.230541  3492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.125:45183: connect: Connection refused (error 111)
18:50:08.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.232522  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.125:45183: Network error: Client connection negotiation failed: client connection to 127.3.97.125:45183: connect: Connection refused (error 111)
18:50:08.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.281872  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } attempt: 1
18:50:08.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.284246  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.125:45183: Network error: Client connection negotiation failed: client connection to 127.3.97.125:45183: connect: Connection refused (error 111)
18:50:08.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.352552  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } attempt: 2
18:50:08.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.354995  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.125:45183: Network error: Client connection negotiation failed: client connection to 127.3.97.125:45183: connect: Connection refused (error 111)
18:50:08.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.446331  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } attempt: 3
18:50:08.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.449045  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.125:45183: Network error: Client connection negotiation failed: client connection to 127.3.97.125:45183: connect: Connection refused (error 111)
18:50:08.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.568486  3556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:08.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.568825  3556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:08.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.568917  3556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:08.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.576859  3556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
18:50:08.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.577008  3556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:08.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.577054  3556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
18:50:08.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.577106  3556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
18:50:08.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.577145  3556 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
18:50:08.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.577207  3556 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:43897/jwks.json
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.587599  3556 master_runner.cc:386] Master server non-default flags:
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/wal
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:50:08.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.125:45183
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43897/jwks.json
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data/info.pb
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.125
18:50:08.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/logs
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:08.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:50:08.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:08.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.589210  3556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.591234  3556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:08.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.592422  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } attempt: 4
18:50:08.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.594578  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.125:45183: Network error: Client connection negotiation failed: client connection to 127.3.97.125:45183: connect: Connection refused (error 111)
18:50:08.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.604674  3565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:08.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.604635  3567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:08.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.604751  3564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:08.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.605418  3556 server_base.cc:1034] running on GCE node
18:50:08.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.606451  3556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:08.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.608079  3556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.609364  3556 hybrid_clock.cc:648] HybridClock initialized: now 1786474208609292 us; error 35 us; skew 500 ppm
18:50:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.609833  3556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:08.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.612684  3556 webserver.cc:460] Webserver started at http://127.3.97.125:43033/ using document root <none> and password file <none>
18:50:08.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.613534  3556 fs_manager.cc:359] Metadata directory not provided
18:50:08.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.613713  3556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:08.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.614144  3556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
18:50:08.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.617280  3556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data/instance:
18:50:08.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e532a7a2d48945dfbeb0a425cab9f3e1"
18:50:08.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:08 on dist-test-slave-pgkr"
18:50:08.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.618326  3556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/wal/instance:
18:50:08.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e532a7a2d48945dfbeb0a425cab9f3e1"
18:50:08.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:08 on dist-test-slave-pgkr"
18:50:08.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.624876  3556 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
18:50:08.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.628885  3575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:08.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.630523  3556 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.630780  3556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/wal
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e532a7a2d48945dfbeb0a425cab9f3e1"
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:08 on dist-test-slave-pgkr"
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.630996  3556 fs_report.cc:383] FS layout report
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/wal
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/wal
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data/data
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:08.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:08.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:08.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.645023  3556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:08.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.645871  3556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:08.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.646143  3556 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:08.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.666194  3556 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.125:45183
18:50:08.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.666226  3631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.125:45183 every 8 connection(s)
18:50:08.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.667981  3556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/data/info.pb
18:50:08.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.671525  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 3556
18:50:08.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.671681  3461 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-1/wal/instance
18:50:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.672431  3634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:50:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.672634  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/wal
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/logs
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data/info.pb
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:08.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.124:42301
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.124
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:08.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:08.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:08.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:08.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:50:08.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:08.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:50:08.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43897/jwks.json with env {}
18:50:08.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.681666  3634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } has no permanent_uuid. Determining permanent_uuid...
18:50:08.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.699075  3634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } has no permanent_uuid. Determining permanent_uuid...
18:50:08.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.709748  3634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } has no permanent_uuid. Determining permanent_uuid...
18:50:08.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.711117  3579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:08.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.712915  3634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:08.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.762290  3634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 1
18:50:08.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.764616  3634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:08.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.832935  3634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 2
18:50:08.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.835449  3634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:08.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.893931  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } attempt: 5
18:50:08.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.902283  3551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } has no permanent_uuid. Determining permanent_uuid...
18:50:08.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.904639  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:08.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.926760  3634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 3
18:50:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.929208  3634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:08.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:08.955942  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 1
18:50:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:08.958323  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:09.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.026721  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 2
18:50:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.028761  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:09.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.059195  3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:09.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.059530  3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:09.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.059619  3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:09.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.067193  3636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
18:50:09.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.067314  3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:09.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.067358  3636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
18:50:09.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.067409  3636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
18:50:09.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.067448  3636 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
18:50:09.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.067508  3636 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:43897/jwks.json
18:50:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.072535  3634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 4
18:50:09.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.074739  3634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.078876  3636 master_runner.cc:386] Master server non-default flags:
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/wal
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.124:42301
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43897/jwks.json
18:50:09.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data/info.pb
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.124
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/logs
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:09.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:09.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:09.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.080595  3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:09.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.083257  3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:09.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.096098  3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:09.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.096113  3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:09.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.096292  3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:09.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.097608  3636 server_base.cc:1034] running on GCE node
18:50:09.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.098547  3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:09.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.100195  3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:09.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.101438  3636 hybrid_clock.cc:648] HybridClock initialized: now 1786474209101354 us; error 56 us; skew 500 ppm
18:50:09.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.101792  3636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:09.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.104278  3636 webserver.cc:460] Webserver started at http://127.3.97.124:40367/ using document root <none> and password file <none>
18:50:09.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.104987  3636 fs_manager.cc:359] Metadata directory not provided
18:50:09.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.105099  3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:09.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.105430  3636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
18:50:09.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.108350  3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data/instance:
18:50:09.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2472d4961b1451e9c9b5da3a7be7036"
18:50:09.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:09 on dist-test-slave-pgkr"
18:50:09.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.109122  3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/wal/instance:
18:50:09.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2472d4961b1451e9c9b5da3a7be7036"
18:50:09.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:09 on dist-test-slave-pgkr"
18:50:09.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.114856  3636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
18:50:09.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.118830  3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.120303  3636 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.120469  3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/wal
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2472d4961b1451e9c9b5da3a7be7036"
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:09 on dist-test-slave-pgkr"
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.120679  3636 fs_report.cc:383] FS layout report
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/wal
18:50:09.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/wal
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data/data
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:09.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:09.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.135023  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 3
18:50:09.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.137281  3551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.97.124:42301: Network error: Client connection negotiation failed: client connection to 127.3.97.124:42301: connect: Connection refused (error 111)
18:50:09.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.137326  3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:09.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.138291  3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:09.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.138572  3636 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:09.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.158931  3636 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.124:42301
18:50:09.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.159015  3716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.124:42301 every 8 connection(s)
18:50:09.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.160672  3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/data/info.pb
18:50:09.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.165156  3717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
18:50:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.166867  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 3636
18:50:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.167052  3461 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/master-2/wal/instance
18:50:09.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.173663  3717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } has no permanent_uuid. Determining permanent_uuid...
18:50:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.192888  3717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } has no permanent_uuid. Determining permanent_uuid...
18:50:09.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.206286  3717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } has no permanent_uuid. Determining permanent_uuid...
18:50:09.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.218456  3717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036: Bootstrap starting.
18:50:09.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.223865  3717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036: Neither blocks nor log segments found. Creating new log.
18:50:09.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.226788  3717 log.cc:824] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036: Log is configured to *not* fsync() on all Append() calls
18:50:09.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.231820  3717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036: No bootstrap required, opened a new log
18:50:09.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.241626  3717 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.242038  3717 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:50:09.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.242132  3717 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2472d4961b1451e9c9b5da3a7be7036, State: Initialized, Role: FOLLOWER
18:50:09.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.242915  3717 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.244961  3726 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.245904  3726 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: This master's current role is: FOLLOWER
18:50:09.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.246423  3717 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: configured and running, proceeding with master startup.
18:50:09.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.265201  3738 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
18:50:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.269783  3738 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
18:50:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.262635  3636 master_runner.cc:186] Error getting master registration for 127.3.97.126:46829: OK, instance_id { permanent_uuid: "6a547507249f4364acc4c841f513a707" instance_seqno: 1786474208190281 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
18:50:09.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.283489  3636 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
18:50:09.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.314567  3551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 4
18:50:09.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.331545  3551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: Bootstrap starting.
18:50:09.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.336760  3551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: Neither blocks nor log segments found. Creating new log.
18:50:09.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.338526  3551 log.cc:824] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: Log is configured to *not* fsync() on all Append() calls
18:50:09.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.342541  3551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: No bootstrap required, opened a new log
18:50:09.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.356067  3551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.356614  3551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:50:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.356752  3551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a547507249f4364acc4c841f513a707, State: Initialized, Role: FOLLOWER
18:50:09.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.357681  3551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.362872  3741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.365310  3741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [sys.catalog]: This master's current role is: FOLLOWER
18:50:09.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.365823  3551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [sys.catalog]: configured and running, proceeding with master startup.
18:50:09.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.374025  3634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } attempt: 5
18:50:09.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.395032  3662 tablet.cc:2369] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:50:09.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.399690  3634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: Bootstrap starting.
18:50:09.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.400539  3726 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:09.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.400908  3726 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.402842  3726 leader_election.cc:288] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a547507249f4364acc4c841f513a707 (127.3.97.126:46829), e532a7a2d48945dfbeb0a425cab9f3e1 (127.3.97.125:45183)
18:50:09.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.404309  3522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2472d4961b1451e9c9b5da3a7be7036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a547507249f4364acc4c841f513a707" is_pre_election: true
18:50:09.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.405062  3522 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2472d4961b1451e9c9b5da3a7be7036 in term 0.
18:50:09.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.405907  3656 leader_election.cc:302] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a547507249f4364acc4c841f513a707, b2472d4961b1451e9c9b5da3a7be7036; no voters: 
18:50:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.408914  3726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Leader pre-election won for term 1
18:50:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.409138  3726 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
18:50:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.409267  3726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 0 FOLLOWER]: Advancing to term 1
18:50:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.413509  3726 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.415167  3522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2472d4961b1451e9c9b5da3a7be7036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a547507249f4364acc4c841f513a707"
18:50:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.415448  3522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [term 0 FOLLOWER]: Advancing to term 1
18:50:09.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.426199  3522 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2472d4961b1451e9c9b5da3a7be7036 in term 1.
18:50:09.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.428156  3656 leader_election.cc:302] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a547507249f4364acc4c841f513a707, b2472d4961b1451e9c9b5da3a7be7036; no voters: 
18:50:09.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.428984  3496 tablet.cc:2369] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:50:09.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.438798  3726 leader_election.cc:288] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [CANDIDATE]: Term 1 election: Requested vote from peers 6a547507249f4364acc4c841f513a707 (127.3.97.126:46829), e532a7a2d48945dfbeb0a425cab9f3e1 (127.3.97.125:45183)
18:50:09.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.445466  3726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 1 FOLLOWER]: Leader election won for term 1
18:50:09.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.447275  3726 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [term 1 LEADER]: Becoming Leader. State: Replica: b2472d4961b1451e9c9b5da3a7be7036, State: Running, Role: LEADER
18:50:09.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.452137  3726 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.459785  3634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: Neither blocks nor log segments found. Creating new log.
18:50:09.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.469043  3634 log.cc:824] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: Log is configured to *not* fsync() on all Append() calls
18:50:09.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.476392  3634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: No bootstrap required, opened a new log
18:50:09.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.481894  3750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2472d4961b1451e9c9b5da3a7be7036. Latest consensus state: current_term: 1 leader_uuid: "b2472d4961b1451e9c9b5da3a7be7036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.482509  3750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: This master's current role is: LEADER
18:50:09.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.492308  3758 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
18:50:09.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.496803  3756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
18:50:09.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.499923  3756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
18:50:09.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.503638  3634 raft_consensus.cc:348] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.508591  3634 raft_consensus.cc:374] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:50:09.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.510943  3634 raft_consensus.cc:729] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e532a7a2d48945dfbeb0a425cab9f3e1, State: Initialized, Role: FOLLOWER
18:50:09.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.497828  3475 master_runner.cc:186] Error getting master registration for 127.3.97.125:45183: OK, instance_id { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" instance_seqno: 1786474208660569 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
18:50:09.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.512157  3475 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
18:50:09.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.528565  3634 consensus_queue.cc:260] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } }
18:50:09.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.539317  3760 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.541287  3760 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [sys.catalog]: This master's current role is: FOLLOWER
18:50:09.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.545133  3604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2472d4961b1451e9c9b5da3a7be7036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1"
18:50:09.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.545027  3634 sys_catalog.cc:564] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [sys.catalog]: configured and running, proceeding with master startup.
18:50:09.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.545904  3604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [term 0 FOLLOWER]: Advancing to term 1
18:50:09.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.545064  3605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2472d4961b1451e9c9b5da3a7be7036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" is_pre_election: true
18:50:09.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.551822  3604 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2472d4961b1451e9c9b5da3a7be7036 in term 1.
18:50:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.564329  3604 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [term 1 FOLLOWER]: Refusing update from remote peer b2472d4961b1451e9c9b5da3a7be7036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:50:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.564419  3522 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [term 1 FOLLOWER]: Refusing update from remote peer b2472d4961b1451e9c9b5da3a7be7036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:50:09.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.565402  3750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:50:09.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.566408  3726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:50:09.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.597433  3741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2472d4961b1451e9c9b5da3a7be7036. Latest consensus state: current_term: 1 leader_uuid: "b2472d4961b1451e9c9b5da3a7be7036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.597908  3741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [sys.catalog]: This master's current role is: FOLLOWER
18:50:09.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.601446  3759 mvcc.cc:205] Tried to move back new op lower bound from 7317398362327572480 to 7317398361969065984. Current Snapshot: MvccSnapshot[applied={T|T < 7317398362327572480}]
18:50:09.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.603144  3760 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2472d4961b1451e9c9b5da3a7be7036. Latest consensus state: current_term: 1 leader_uuid: "b2472d4961b1451e9c9b5da3a7be7036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.603493  3760 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [sys.catalog]: This master's current role is: FOLLOWER
18:50:09.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.607065  3741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2472d4961b1451e9c9b5da3a7be7036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.607391  3741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707 [sys.catalog]: This master's current role is: FOLLOWER
18:50:09.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.617481  3726 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2472d4961b1451e9c9b5da3a7be7036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.618189  3726 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: This master's current role is: LEADER
18:50:09.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.618687  3726 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2472d4961b1451e9c9b5da3a7be7036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.619148  3726 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036 [sys.catalog]: This master's current role is: LEADER
18:50:09.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.620823  3756 catalog_manager.cc:1347] Generated new cluster ID: 4ba6686d98f244f3bd48775c414e9e76
18:50:09.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.620951  3756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
18:50:09.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.623227  3760 sys_catalog.cc:455] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2472d4961b1451e9c9b5da3a7be7036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a547507249f4364acc4c841f513a707" member_type: VOTER last_known_addr { host: "127.3.97.126" port: 46829 } } peers { permanent_uuid: "e532a7a2d48945dfbeb0a425cab9f3e1" member_type: VOTER last_known_addr { host: "127.3.97.125" port: 45183 } } peers { permanent_uuid: "b2472d4961b1451e9c9b5da3a7be7036" member_type: VOTER last_known_addr { host: "127.3.97.124" port: 42301 } } }
18:50:09.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.623528  3760 sys_catalog.cc:458] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1 [sys.catalog]: This master's current role is: FOLLOWER
18:50:09.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.655226  3782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
18:50:09.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.655740  3782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
18:50:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:09.663633  3656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.97.126:46829, user_credentials={real_user=slave}} blocked reactor thread for 56326.8us
18:50:09.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.668300  3756 catalog_manager.cc:1370] Generated new certificate authority record
18:50:09.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.670166  3756 catalog_manager.cc:1504] Loading token signing keys...
18:50:09.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.683624  3756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2472d4961b1451e9c9b5da3a7be7036: Generated new TSK 0
18:50:09.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.684767  3756 catalog_manager.cc:1514] Initializing in-progress tserver states...
18:50:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:09.698891  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:09.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:0
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:09.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:10.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.091730  3787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:10.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.092109  3787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:10.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.092278  3787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:10.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.100679  3787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:10.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.100981  3787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.65
18:50:10.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.111302  3787 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:10.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:10.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:10.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:0
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:10.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:10.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:10.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:10.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:10.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.113258  3787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:10.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.115396  3787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:10.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.130012  3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.131152  3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.131444  3787 server_base.cc:1034] running on GCE node
18:50:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.131819  3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.132846  3787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.134414  3787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:10.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.135975  3787 hybrid_clock.cc:648] HybridClock initialized: now 1786474210135832 us; error 125 us; skew 500 ppm
18:50:10.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.136404  3787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:10.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.143018  3787 webserver.cc:460] Webserver started at http://127.3.97.65:38971/ using document root <none> and password file <none>
18:50:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.144089  3787 fs_manager.cc:359] Metadata directory not provided
18:50:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.144259  3787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:10.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.144637  3787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
18:50:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.148856  3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/instance:
18:50:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "32aab08028a947089da244a350b00c7e"
18:50:10.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.150089  3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal/instance:
18:50:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "32aab08028a947089da244a350b00c7e"
18:50:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:10.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.157763  3787 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
18:50:10.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.161875  3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:10.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.164572  3787 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.003s
18:50:10.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.165194  3787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:10.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "32aab08028a947089da244a350b00c7e"
18:50:10.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:10.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.165438  3787 fs_report.cc:383] FS layout report
18:50:10.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/data
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:10.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:10.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.180974  3787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:10.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.182073  3787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:10.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.182461  3787 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:10.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.183706  3787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:10.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.185940  3787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:50:10.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.186079  3787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:50:10.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.186239  3787 ts_tablet_manager.cc:610] Registered 0 tablets
18:50:10.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.186317  3787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:50:10.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.226604  3787 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.65:42129
18:50:10.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.226709  3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.65:42129 every 8 connection(s)
18:50:10.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.228367  3787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.235354  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 3787
18:50:10.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.235647  3461 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal/instance
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.243191  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:0
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:10.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:10.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.257906  3936 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:10.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.258754  3936 heartbeater.cc:464] Registering TS with master...
18:50:10.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.258921  3933 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:10.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.259552  3933 heartbeater.cc:464] Registering TS with master...
18:50:10.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.260675  3936 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:10.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.260701  3933 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.263969  3593 ts_manager.cc:194] Registered new tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.265255  3934 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.265388  3934 heartbeater.cc:464] Registering TS with master...
18:50:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.265714  3675 ts_manager.cc:194] Registered new tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.265914  3934 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.267776  3509 ts_manager.cc:194] Registered new tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:10.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.268144  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.65:56457
18:50:10.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.498350  3758 catalog_manager.cc:1259] Loaded cluster ID: 4ba6686d98f244f3bd48775c414e9e76
18:50:10.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.498513  3758 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: loading cluster ID for follower catalog manager: success
18:50:10.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.502497  3758 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: acquiring CA information for follower catalog manager: success
18:50:10.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.505262  3758 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a547507249f4364acc4c841f513a707: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
18:50:10.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.643381  3944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.643759  3944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:10.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.643884  3944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:10.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.652693  3944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:10.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.652979  3944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.66
18:50:10.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.659350  3782 catalog_manager.cc:1259] Loaded cluster ID: 4ba6686d98f244f3bd48775c414e9e76
18:50:10.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.659487  3782 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: loading cluster ID for follower catalog manager: success
18:50:10.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.663616  3782 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: acquiring CA information for follower catalog manager: success
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.664199  3944 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:0
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:10.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:10.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:10.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.667138  3782 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e532a7a2d48945dfbeb0a425cab9f3e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
18:50:10.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.667303  3944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:10.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.669811  3944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:10.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.692606  3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:10.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.693801  3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:10.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:10.695308  3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:10.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.696419  3944 server_base.cc:1034] running on GCE node
18:50:10.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.697542  3944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.699043  3944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:10.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.702558  3944 hybrid_clock.cc:648] HybridClock initialized: now 1786474210702457 us; error 89 us; skew 500 ppm
18:50:10.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.703091  3944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:10.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.706544  3944 webserver.cc:460] Webserver started at http://127.3.97.66:36651/ using document root <none> and password file <none>
18:50:10.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.707588  3944 fs_manager.cc:359] Metadata directory not provided
18:50:10.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.707782  3944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:10.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.708169  3944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
18:50:10.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.712376  3944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/instance:
18:50:10.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:10.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:10.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.713546  3944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal/instance:
18:50:10.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:10.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:10.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.721290  3944 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.009s	sys 0.000s
18:50:10.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.726955  3963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.729059  3944 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
18:50:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.729303  3944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.729581  3944 fs_report.cc:383] FS layout report
18:50:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/data
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:10.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:10.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.751362  3944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:10.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.752796  3944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:10.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.753218  3944 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:10.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.754786  3944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:10.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.757247  3944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:50:10.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.757367  3944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:50:10.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.757514  3944 ts_tablet_manager.cc:610] Registered 0 tablets
18:50:10.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.757589  3944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:50:10.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.835709  3944 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.66:42625
18:50:10.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.835803  4094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.66:42625 every 8 connection(s)
18:50:10.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.837857  3944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:10.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.843073  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 3944
18:50:10.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.843268  3461 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal/instance
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.857931  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:10.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:0
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:10.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:10.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:10.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:10.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:10.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:10.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:10.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.906956  4097 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:10.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.907377  4097 heartbeater.cc:464] Registering TS with master...
18:50:10.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.908444  4097 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:10.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.910571  3509 ts_manager.cc:194] Registered new tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:10.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.912263  4095 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:10.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.912387  4095 heartbeater.cc:464] Registering TS with master...
18:50:10.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.912837  4095 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:10.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.914093  3675 ts_manager.cc:194] Registered new tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:10.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.915329  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.66:34245
18:50:10.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.916155  4098 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:10.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.916309  4098 heartbeater.cc:464] Registering TS with master...
18:50:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.916877  4098 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:10.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:10.918852  3593 ts_manager.cc:194] Registered new tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:11.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.271595  3933 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.297886  4106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.298264  4106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:11.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.298393  4106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:11.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.306239  4106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:11.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.306504  4106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.67
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.316687  4106 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:0
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:11.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:11.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.318548  4106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:11.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.320560  4106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.334903  4113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.335356  4114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:11.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:11.335707  4117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:11.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.336544  4106 server_base.cc:1034] running on GCE node
18:50:11.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.337284  4106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:11.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.338375  4106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:11.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.339627  4106 hybrid_clock.cc:648] HybridClock initialized: now 1786474211339524 us; error 74 us; skew 500 ppm
18:50:11.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.339977  4106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:11.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.342417  4106 webserver.cc:460] Webserver started at http://127.3.97.67:40339/ using document root <none> and password file <none>
18:50:11.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.343189  4106 fs_manager.cc:359] Metadata directory not provided
18:50:11.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.343339  4106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:11.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.343679  4106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
18:50:11.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.346480  4106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/instance:
18:50:11.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:11.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:11 on dist-test-slave-pgkr"
18:50:11.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.347422  4106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal/instance:
18:50:11.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:11.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:11 on dist-test-slave-pgkr"
18:50:11.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.352943  4106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
18:50:11.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.356925  4124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:11.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.358485  4106 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
18:50:11.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.358700  4106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:11.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:11.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:11 on dist-test-slave-pgkr"
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.358893  4106 fs_report.cc:383] FS layout report
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/data
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:11.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:11.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.401012  4106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:11.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.401902  4106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:11.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.402177  4106 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:11.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.403278  4106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:11.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.404943  4106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
18:50:11.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.405019  4106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
18:50:11.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.405113  4106 ts_tablet_manager.cc:610] Registered 0 tablets
18:50:11.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.405151  4106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
18:50:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.440276  4106 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.67:39649
18:50:11.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.440300  4250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.67:39649 every 8 connection(s)
18:50:11.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.441521  4106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:11.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.449111  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 4106
18:50:11.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.449373  3461 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal/instance
18:50:11.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.499572  4251 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:11.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.500005  4251 heartbeater.cc:464] Registering TS with master...
18:50:11.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.500988  4251 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:11.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.501757  4253 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:11.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.501897  4253 heartbeater.cc:464] Registering TS with master...
18:50:11.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.502370  4253 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:11.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.502712  4252 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:11.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.502697  3675 ts_manager.cc:194] Registered new tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:11.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.502839  4252 heartbeater.cc:464] Registering TS with master...
18:50:11.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.503211  4252 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.503796  3592 ts_manager.cc:194] Registered new tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.503873  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.67:54767
18:50:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.504433  3509 ts_manager.cc:194] Registered new tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:11.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.514973  3461 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
18:50:11.516 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
18:50:11.518 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

18:50:11.528 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.3.97.126"
      port: 46829
    }
    bound_http_address {
      host: "127.3.97.126"
      port: 33553
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.3.97.125"
      port: 45183
    }
    bound_http_address {
      host: "127.3.97.125"
      port: 43033
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.3.97.124"
      port: 42301
    }
    bound_http_address {
      host: "127.3.97.124"
      port: 40367
    }
  }
}

18:50:11.534 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

18:50:11.537 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.3.97.65"
      port: 42129
    }
    bound_http_address {
      host: "127.3.97.65"
      port: 38971
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.3.97.66"
      port: 42625
    }
    bound_http_address {
      host: "127.3.97.66"
      port: 36651
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.3.97.67"
      port: 39649
    }
    bound_http_address {
      host: "127.3.97.67"
      port: 40339
    }
  }
}

18:50:11.537 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
18:50:11.726 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
18:50:11.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:11.918592  4095 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:11.920 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.97.124:42301(127.3.97.124:42301)] connecting to peer
18:50:11.978 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.97.125:45183(127.3.97.125:45183)] connecting to peer
18:50:11.982 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.3.97.126:46829(127.3.97.126:46829)] connecting to peer
18:50:12.019 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.3.97.125:45183(127.3.97.125:45183)] Successfully connected to peer
18:50:12.022 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.3.97.124:42301(127.3.97.124:42301)] Successfully connected to peer
18:50:12.093 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.3.97.126:46829(127.3.97.126:46829)] Successfully connected to peer
18:50:12.120 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.3.97.125
18:50:12.120 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.3.97.124
18:50:12.243 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.3.97.126
18:50:12.371 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x21fb2550, L:/127.0.0.1:59442 - R:/127.3.97.125:45183] using SASL/PLAIN
18:50:12.372 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x36223c36, L:/127.0.0.1:52080 - R:/127.3.97.124:42301] using SASL/PLAIN
18:50:12.415 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
18:50:12.417 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=539), Traces: [0ms] refreshing cache from master, [30ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.97.124:42301, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.97.125:45183, [101ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.97.126:46829, [505ms] Sub RPC ConnectToMaster: received response from server master-127.3.97.125:45183: OK, [506ms] Sub RPC ConnectToMaster: received response from server master-127.3.97.124:42301: OK, deferred=null) after lookup
18:50:12.472 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x15f0115f, L:/127.0.0.1:37110 - R:/127.3.97.126:46829] using SASL/PLAIN
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.469966  3675 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52080:
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865"
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:50:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
18:50:12.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
18:50:12.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:12.472344  3675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
18:50:12.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.506274  4251 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:12.512 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table d58f08ebcefe46e0a955c17a05271433
18:50:12.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.514832  3857 tablet_service.cc:1467] Processing CreateTablet for tablet c92f0a97947e4a8e8362a7d223996d8b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865 [id=d58f08ebcefe46e0a955c17a05271433]), partition=RANGE (key) PARTITION UNBOUNDED
18:50:12.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.517184  3857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92f0a97947e4a8e8362a7d223996d8b. 1 dirs total, 0 dirs full, 0 dirs failed
18:50:12.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.531291  4016 tablet_service.cc:1467] Processing CreateTablet for tablet c92f0a97947e4a8e8362a7d223996d8b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865 [id=d58f08ebcefe46e0a955c17a05271433]), partition=RANGE (key) PARTITION UNBOUNDED
18:50:12.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.533279  4016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92f0a97947e4a8e8362a7d223996d8b. 1 dirs total, 0 dirs full, 0 dirs failed
18:50:12.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.533957  4290 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap starting.
18:50:12.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.534164  4176 tablet_service.cc:1467] Processing CreateTablet for tablet c92f0a97947e4a8e8362a7d223996d8b (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865 [id=d58f08ebcefe46e0a955c17a05271433]), partition=RANGE (key) PARTITION UNBOUNDED
18:50:12.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.536036  4176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c92f0a97947e4a8e8362a7d223996d8b. 1 dirs total, 0 dirs full, 0 dirs failed
18:50:12.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.538772  4290 tablet_bootstrap.cc:654] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Neither blocks nor log segments found. Creating new log.
18:50:12.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.540264  4290 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Log is configured to *not* fsync() on all Append() calls
18:50:12.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.556836  4292 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap starting.
18:50:12.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.557124  4290 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: No bootstrap required, opened a new log
18:50:12.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.557446  4290 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.013s
18:50:12.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.561044  4292 tablet_bootstrap.cc:654] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Neither blocks nor log segments found. Creating new log.
18:50:12.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.563395  4292 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Log is configured to *not* fsync() on all Append() calls
18:50:12.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.564958  4293 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap starting.
18:50:12.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.570127  4290 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.570856  4290 raft_consensus.cc:374] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:50:12.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.570983  4290 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aab08028a947089da244a350b00c7e, State: Initialized, Role: FOLLOWER
18:50:12.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.571765  4290 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.572769  4292 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: No bootstrap required, opened a new log
18:50:12.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.573805  4290 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent starting tablet: real 0.016s	user 0.014s	sys 0.000s
18:50:12.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.574445  4292 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent bootstrapping tablet: real 0.018s	user 0.003s	sys 0.012s
18:50:12.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.578560  4293 tablet_bootstrap.cc:654] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Neither blocks nor log segments found. Creating new log.
18:50:12.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.582372  4293 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Log is configured to *not* fsync() on all Append() calls
18:50:12.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.586081  4293 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: No bootstrap required, opened a new log
18:50:12.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.586429  4293 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent bootstrapping tablet: real 0.022s	user 0.008s	sys 0.000s
18:50:12.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.587517  4292 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.588057  4292 raft_consensus.cc:374] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:50:12.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.588173  4292 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Initialized, Role: FOLLOWER
18:50:12.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.588848  4292 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.590971  4292 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent starting tablet: real 0.016s	user 0.012s	sys 0.002s
18:50:12.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.597927  4293 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.598341  4293 raft_consensus.cc:374] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
18:50:12.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.598413  4293 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa44f6302c4b4e11adf404ce13329e9d, State: Initialized, Role: FOLLOWER
18:50:12.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.599028  4293 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.601212  4293 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.010s
18:50:12.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.614187  4295 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:12.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.614545  4295 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.616309  4295 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:12.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:12.624114  4105 tablet.cc:2369] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:50:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.625435  4042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b" is_pre_election: true
18:50:12.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.626161  4042 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 0.
18:50:12.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.626785  4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:12.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.627108  3806 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b; no voters: 
18:50:12.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.627303  4198 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 0.
18:50:12.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.627507  4295 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Leader pre-election won for term 1
18:50:12.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.627698  4295 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
18:50:12.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.627835  4295 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 0 FOLLOWER]: Advancing to term 1
18:50:12.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.629928  4295 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.630636  4295 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 1 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:12.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.631132  4042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:12.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.631232  4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:12.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.631362  4042 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 0 FOLLOWER]: Advancing to term 1
18:50:12.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.631417  4198 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 0 FOLLOWER]: Advancing to term 1
18:50:12.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.633571  4198 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 1.
18:50:12.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.633795  4042 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 1.
18:50:12.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.634017  3807 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d; no voters: 
18:50:12.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.634399  4295 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Leader election won for term 1
18:50:12.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.634979  4295 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 LEADER]: Becoming Leader. State: Replica: 32aab08028a947089da244a350b00c7e, State: Running, Role: LEADER
18:50:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.635625  4295 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:12.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.642328  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 32aab08028a947089da244a350b00c7e (127.3.97.65). New cstate: current_term: 1 leader_uuid: "32aab08028a947089da244a350b00c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:12.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:12.695916  4254 tablet.cc:2369] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:50:12.705 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table d58f08ebcefe46e0a955c17a05271433
18:50:12.727 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:12.734 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299993, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:12.736 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [9ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=null) after lookup
18:50:12.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:12.739423  3937 tablet.cc:2369] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
18:50:12.747 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:12.749 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] Successfully connected to peer
18:50:12.798 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x73da7b91, L:/127.0.0.1:47386 - R:/127.3.97.65:42129] using TOKEN/null
18:50:12.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.808820  4198 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Refusing update from remote peer 32aab08028a947089da244a350b00c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:50:12.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.808822  4042 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Refusing update from remote peer 32aab08028a947089da244a350b00c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
18:50:12.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.810196  4302 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:50:12.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.810669  4295 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
18:50:12.938 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:12.941 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.65:42129
18:50:12.942 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:12.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.943236  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 3787
18:50:12.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:12.953881  3927 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:13.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:13.055080  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 3787
18:50:13.083 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:13.087 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] handling channelInactive
18:50:13.087 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state READY due to: connection closed
18:50:13.088 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: connection closed
18:50:13.103 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:13.103 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:13.104 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [4ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@416771488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@88f35a5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@501343de -> wakeup thread Time-limited test)) after lookup
18:50:13.105 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:13.106 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.106 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.106 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.123 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:13.123 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:13.124 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [4ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [20ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [22ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@416771488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@88f35a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@501343de -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306))) after lookup
18:50:13.125 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:13.127 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.127 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.128 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.163 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:13.163 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:13.164 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [4ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [20ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [22ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [78ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@416771488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@88f35a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@501343de -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466))) after lookup
18:50:13.165 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:13.166 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.166 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.166 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.283 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:13.283 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:13.284 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=199), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [4ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [20ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [22ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [78ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [80ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [82ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [82ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [198ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@416771488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@88f35a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@501343de -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901))) after lookup
18:50:13.285 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:13.287 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.287 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.288 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.722 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:13.722 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:13.723 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=638), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [4ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [20ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [22ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [78ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [80ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [82ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [82ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [198ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [201ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [204ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [204ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [634ms] refreshing cache from master, [634ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [637ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@416771488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@88f35a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901) -> (continuation of Deferred@86403220 after retry RPC@2113117902), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@501343de -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901) -> (continuation of Deferred@86403220 after retry RPC@2113117902))) after lookup
18:50:13.724 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:13.725 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.725 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:13.725 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:14.043 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:14.044 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:14.045 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=960), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [4ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [20ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [22ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [78ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [80ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [82ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [82ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [198ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [201ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [204ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [204ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [634ms] refreshing cache from master, [634ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [637ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [640ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [641ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [642ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [955ms] refreshing cache from master, [955ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [958ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@416771488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@88f35a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901) -> (continuation of Deferred@86403220 after retry RPC@2113117902) -> (continuation of Deferred@1439267849 after retry RPC@1015603295), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@501343de -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901) -> (continuation of Deferred@86403220 after retry RPC@2113117902) -> (continuation of Deferred@1439267849 after retry RPC@1015603295))) after lookup
18:50:14.046 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:14.048 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:14.048 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:14.048 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:14.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.387439  4321 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:14.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.387710  4321 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:14.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.388283  4322 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:14.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.388450  4322 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:14.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.389528  4321 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:14.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.389698  4322 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:14.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:14.391234  4127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:14.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:14.393603  3966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:14.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:14.393712  4127 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:14.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:14.395644  3966 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:14.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.399873  4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:14.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.400169  4198 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 1.
18:50:14.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.400333  4042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b" is_pre_election: true
18:50:14.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.400627  4042 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa44f6302c4b4e11adf404ce13329e9d in term 1.
18:50:14.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.400715  3966 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:14.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.401124  4322 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Leader pre-election won for term 2
18:50:14.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.401329  4322 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:14.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.401477  4322 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 1 FOLLOWER]: Advancing to term 2
18:50:14.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.401499  4126 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:14.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.401894  4321 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Leader pre-election won for term 2
18:50:14.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.402027  4321 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:14.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.402151  4321 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 1 FOLLOWER]: Advancing to term 2
18:50:14.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.403831  4322 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:14.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.404196  4321 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:14.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.404538  4322 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 2 election: Requested vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:14.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.405346  4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:14.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.405347  4042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:14.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.405638  4198 raft_consensus.cc:2379] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 857be9840834437fb311c99b57bf0b4b in current term 2: Already voted for candidate fa44f6302c4b4e11adf404ce13329e9d in this term.
18:50:14.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.405644  4042 raft_consensus.cc:2379] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fa44f6302c4b4e11adf404ce13329e9d in current term 2: Already voted for candidate 857be9840834437fb311c99b57bf0b4b in this term.
18:50:14.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.405769  4321 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 2 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:14.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:14.406651  4127 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:14.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.406806  4127 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b
18:50:14.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.407100  4321 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
18:50:14.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:14.407325  3966 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:14.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.407500  3966 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b; no voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d
18:50:14.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:14.407840  4322 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
18:50:16.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.103101  4332 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:16.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.103308  4332 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:16.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.103883  4332 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:16.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.105336  4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:16.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.105638  4198 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 2.
18:50:16.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.106148  3966 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 
18:50:16.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:16.106402  3966 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:16.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.106455  4332 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Leader pre-election won for term 3
18:50:16.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.106645  4332 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
18:50:16.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.106771  4332 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 2 FOLLOWER]: Advancing to term 3
18:50:16.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.108736  4332 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.109328  4332 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 3 election: Requested vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.110114  4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:16.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.110360  4198 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 2 FOLLOWER]: Advancing to term 3
18:50:16.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:16.111040  3966 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:16.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.112473  4198 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 3.
18:50:16.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.112944  3966 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:16.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.113266  4332 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 3 FOLLOWER]: Leader election won for term 3
18:50:16.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.113708  4332 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 3 LEADER]: Becoming Leader. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Running, Role: LEADER
18:50:16.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.114338  4332 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:16.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.118786  3676 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b reported cstate change: term changed from 1 to 3, leader changed from 32aab08028a947089da244a350b00c7e (127.3.97.65) to 857be9840834437fb311c99b57bf0b4b (127.3.97.66). New cstate: current_term: 3 leader_uuid: "857be9840834437fb311c99b57bf0b4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:16.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:16.592941  3966 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b -> Peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Couldn't send request to peer 32aab08028a947089da244a350b00c7e. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:16.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.727829  4198 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 3 FOLLOWER]: Refusing update from remote peer 857be9840834437fb311c99b57bf0b4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
18:50:16.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:16.728708  4332 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
18:50:17.102 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:17.103 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:17.104 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4019), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [4ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [18ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [20ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [22ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [78ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [80ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [82ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [82ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [198ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [201ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [204ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [204ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [634ms] refreshing cache from master, [634ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [637ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [640ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [641ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [642ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [955ms] refreshing cache from master, [955ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [958ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [962ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [964ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [964ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [4014ms] refreshing cache from master, [4014ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [4017ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@416771488(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@88f35a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901) -> (continuation of Deferred@86403220 after retry RPC@2113117902) -> (continuation of Deferred@1439267849 after retry RPC@1015603295) -> (continuation of Deferred@1684691135 after retry RPC@530302581), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@501343de -> wakeup thread Time-limited test -> (continuation of Deferred@158812999 after retry RPC@1014814306) -> (continuation of Deferred@2017122401 after retry RPC@573558466) -> (continuation of Deferred@390281603 after retry RPC@2066772901) -> (continuation of Deferred@86403220 after retry RPC@2113117902) -> (continuation of Deferred@1439267849 after retry RPC@1015603295) -> (continuation of Deferred@1684691135 after retry RPC@530302581))) after lookup
18:50:17.106 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:17.107 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] Successfully connected to peer
18:50:17.133 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x06cf36cc, L:/127.0.0.1:47156 - R:/127.3.97.66:42625] using TOKEN/null
18:50:17.159 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.65:42129
18:50:17.161 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.162948  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:17.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:17.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:17.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.524704  4367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:17.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.525050  4367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:17.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.525170  4367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:17.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.533066  4367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:17.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.533344  4367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.65
18:50:17.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.543423  4367 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:17.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:17.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:17.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:17.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:17.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.545193  4367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:17.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.547140  4367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:17.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.562238  4372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:17.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.562270  4377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:17.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.562167  4373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:17.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.563004  4367 server_base.cc:1034] running on GCE node
18:50:17.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.563776  4367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:17.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.564831  4367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:17.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.566130  4367 hybrid_clock.cc:648] HybridClock initialized: now 1786474217566033 us; error 45 us; skew 500 ppm
18:50:17.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.566418  4367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:17.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.569078  4367 webserver.cc:460] Webserver started at http://127.3.97.65:38971/ using document root <none> and password file <none>
18:50:17.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.569780  4367 fs_manager.cc:359] Metadata directory not provided
18:50:17.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.569902  4367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:17.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.575786  4367 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
18:50:17.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.578902  4385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:17.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.580271  4367 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
18:50:17.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.580477  4367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:17.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "32aab08028a947089da244a350b00c7e"
18:50:17.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:17.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.581569  4367 fs_report.cc:383] FS layout report
18:50:17.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:17.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/data
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:17.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:17.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.597836  4367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:17.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.598837  4367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:17.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.599193  4367 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:17.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.600322  4367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:17.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.602450  4392 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:17.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.608071  4367 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:17.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.608248  4367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:50:17.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.608385  4367 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:17.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.611485  4367 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:17.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.611599  4367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.000s	sys 0.003s
18:50:17.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.611856  4392 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap starting.
18:50:17.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.636494  4392 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Log is configured to *not* fsync() on all Append() calls
18:50:17.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.652906  4392 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:17.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.653646  4392 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap complete.
18:50:17.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.655201  4392 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent bootstrapping tablet: real 0.044s	user 0.027s	sys 0.015s
18:50:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.655850  4367 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.65:42129
18:50:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.655887  4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.65:42129 every 8 connection(s)
18:50:17.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.657310  4367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:17.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.664623  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 4367
18:50:17.665 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:17.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.668497  4392 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:17.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.670078  4392 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aab08028a947089da244a350b00c7e, State: Initialized, Role: FOLLOWER
18:50:17.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.671521  4392 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:17.679 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:17.680 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.66:42625
18:50:17.680 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:50:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.680840  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 3944
18:50:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.680876  4392 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent starting tablet: real 0.026s	user 0.011s	sys 0.002s
18:50:17.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.711745  4521 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:17.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.712116  4521 heartbeater.cc:464] Registering TS with master...
18:50:17.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.713227  4521 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:17.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.715770  4520 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:17.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.715909  4520 heartbeater.cc:464] Registering TS with master...
18:50:17.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.716346  4520 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:17.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.716452  3509 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:17.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.717836  3675 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:17.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.719364  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.65:52301
18:50:17.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.720388  4088 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:17.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.720718  4526 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:17.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.720876  4526 heartbeater.cc:464] Registering TS with master...
18:50:17.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.721724  4526 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:17.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.722558  4520 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:17.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.723394  3593 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:17.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:17.826156  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 3944
18:50:17.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.852523  3494 connection.cc:535] server connection from 127.3.97.66:35433 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:50:17.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.852814  3578 connection.cc:535] server connection from 127.3.97.66:46397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:50:17.853 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] handling channelInactive
18:50:17.853 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state READY due to: connection closed
18:50:17.853 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:17.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:17.853173  3656 connection.cc:535] server connection from 127.3.97.66:34245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:50:17.854 [DEBUG - Time-limited test] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:17.855 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.855 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.856 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.861 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:17.862 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:17.863 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [2ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1464030851(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@54ba50a5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58cf08f3 -> wakeup thread Time-limited test)) after lookup
18:50:17.864 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:17.865 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.865 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.865 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.883 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:17.884 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:17.884 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [2ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [10ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [12ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [12ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1464030851(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@54ba50a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58cf08f3 -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836))) after lookup
18:50:17.885 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:17.887 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.887 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.888 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.902 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:17.902 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:17.903 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [2ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [10ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [12ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [12ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [32ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [35ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [35ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1464030851(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@54ba50a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58cf08f3 -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936))) after lookup
18:50:17.904 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:17.905 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.905 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.905 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.982 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:17.982 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:17.983 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=129), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [2ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [10ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [12ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [12ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [32ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [35ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [35ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [51ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [52ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [52ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [128ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1464030851(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@54ba50a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58cf08f3 -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722))) after lookup
18:50:17.984 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:17.986 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.986 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:17.986 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:18.002 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:18.002 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:18.003 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=149), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [2ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [10ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [12ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [12ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [32ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [35ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [35ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [51ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [52ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [52ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [128ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [130ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [133ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [134ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [148ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1464030851(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@54ba50a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722) -> (continuation of Deferred@1264486050 after retry RPC@1317120110), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58cf08f3 -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722) -> (continuation of Deferred@1264486050 after retry RPC@1317120110))) after lookup
18:50:18.004 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:18.005 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:18.005 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:18.005 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:18.942 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:18.942 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:18.943 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1089), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [2ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [10ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [12ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [12ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [32ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [35ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [35ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [51ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [52ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [52ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [128ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [130ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [133ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [134ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [148ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [151ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [152ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [152ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [1085ms] refreshing cache from master, [1085ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [1088ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1464030851(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@54ba50a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722) -> (continuation of Deferred@1264486050 after retry RPC@1317120110) -> (continuation of Deferred@1139308013 after retry RPC@2045911773), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58cf08f3 -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722) -> (continuation of Deferred@1264486050 after retry RPC@1317120110) -> (continuation of Deferred@1139308013 after retry RPC@2045911773))) after lookup
18:50:18.944 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:18.946 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:18.946 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:18.946 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:19.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.174082  4537 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.174556  4537 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:19.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.176838  4537 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:19.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:19.178465  4387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:19.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:19.180508  4387 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:19.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.185753  4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:19.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.186832  4388 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aab08028a947089da244a350b00c7e; no voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d
18:50:19.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.187419  4537 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
18:50:19.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.211215  4543 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 857be9840834437fb311c99b57bf0b4b)
18:50:19.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.211409  4543 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:19.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.212154  4543 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:19.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:19.214838  4126 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:19.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.221025  4467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e" is_pre_election: true
18:50:19.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.221505  4467 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa44f6302c4b4e11adf404ce13329e9d in term 1.
18:50:19.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.222491  4127 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d; no voters: 857be9840834437fb311c99b57bf0b4b
18:50:19.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.222903  4543 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 3 FOLLOWER]: Leader pre-election won for term 4
18:50:19.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.223042  4543 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 857be9840834437fb311c99b57bf0b4b)
18:50:19.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.223120  4543 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 3 FOLLOWER]: Advancing to term 4
18:50:19.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.225349  4543 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:19.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.226195  4543 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 4 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:19.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.226922  4467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e"
18:50:19.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.227195  4467 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 1 FOLLOWER]: Advancing to term 4
18:50:19.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:19.227805  4126 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:19.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.231781  4467 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa44f6302c4b4e11adf404ce13329e9d in term 4.
18:50:19.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.232376  4127 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d; no voters: 857be9840834437fb311c99b57bf0b4b
18:50:19.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.232757  4543 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 4 FOLLOWER]: Leader election won for term 4
18:50:19.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.233268  4543 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 4 LEADER]: Becoming Leader. State: Replica: fa44f6302c4b4e11adf404ce13329e9d, State: Running, Role: LEADER
18:50:19.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.233836  4543 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:19.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.237712  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d reported cstate change: term changed from 3 to 4, leader changed from 857be9840834437fb311c99b57bf0b4b (127.3.97.66) to fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67). New cstate: current_term: 4 leader_uuid: "fa44f6302c4b4e11adf404ce13329e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: HEALTHY } } }
18:50:19.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.741791  4467 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 4 FOLLOWER]: Refusing update from remote peer fa44f6302c4b4e11adf404ce13329e9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
18:50:19.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:19.742821  4564 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
18:50:19.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:19.763402  4126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111) [suppressed 3 similar messages]
18:50:19.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:19.765057  4126 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d -> Peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Couldn't send request to peer 857be9840834437fb311c99b57bf0b4b. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:20.084 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:20.084 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:20.085 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2231), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [2ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [8ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [10ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [12ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [12ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [29ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [32ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [35ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [35ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [48ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [51ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [52ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [52ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [125ms] refreshing cache from master, [125ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [128ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [130ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [133ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [134ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [148ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [151ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [152ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [152ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [1085ms] refreshing cache from master, [1085ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [1088ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [1090ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [1093ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [1093ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [2226ms] refreshing cache from master, [2226ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [2229ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1464030851(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@54ba50a5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722) -> (continuation of Deferred@1264486050 after retry RPC@1317120110) -> (continuation of Deferred@1139308013 after retry RPC@2045911773) -> (continuation of Deferred@851738488 after retry RPC@1514686263), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58cf08f3 -> wakeup thread Time-limited test -> (continuation of Deferred@1582837350 after retry RPC@1314676836) -> (continuation of Deferred@823519318 after retry RPC@476569936) -> (continuation of Deferred@1919321471 after retry RPC@460052722) -> (continuation of Deferred@1264486050 after retry RPC@1317120110) -> (continuation of Deferred@1139308013 after retry RPC@2045911773) -> (continuation of Deferred@851738488 after retry RPC@1514686263))) after lookup
18:50:20.086 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:20.087 [DEBUG - kudu-nio-0] (Connection.java:757) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] Successfully connected to peer
18:50:20.113 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x2d2ce659, L:/127.0.0.1:50354 - R:/127.3.97.67:39649] using TOKEN/null
18:50:20.140 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.66:42625
18:50:20.141 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:50:20.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.141745  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:42625
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36651
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:20.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:20.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.514858  4581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:20.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.515246  4581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:20.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.515374  4581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:20.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.523427  4581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:20.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.523686  4581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.66
18:50:20.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.533380  4581 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:20.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:20.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:20.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:20.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:42625
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36651
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:20.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:20.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.535353  4581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:20.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.537364  4581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:20.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.552788  4590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:20.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.552784  4591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:20.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:20.553004  4593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:20.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.553933  4581 server_base.cc:1034] running on GCE node
18:50:20.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.554812  4581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:20.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.556085  4581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:20.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.557345  4581 hybrid_clock.cc:648] HybridClock initialized: now 1786474220557270 us; error 42 us; skew 500 ppm
18:50:20.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.557716  4581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:20.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.564661  4581 webserver.cc:460] Webserver started at http://127.3.97.66:36651/ using document root <none> and password file <none>
18:50:20.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.565537  4581 fs_manager.cc:359] Metadata directory not provided
18:50:20.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.565727  4581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.571935  4581 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
18:50:20.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.575456  4599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:20.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.577067  4581 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
18:50:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.577319  4581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.578555  4581 fs_report.cc:383] FS layout report
18:50:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/data
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:20.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.613863  4581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:20.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.614883  4581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:20.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.615250  4581 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:20.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.616367  4581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:20.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.618598  4608 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:20.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.624329  4581 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:20.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.624490  4581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:50:20.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.624631  4581 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:20.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.627861  4581 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:20.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.627983  4581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
18:50:20.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.628458  4608 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap starting.
18:50:20.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.652079  4608 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Log is configured to *not* fsync() on all Append() calls
18:50:20.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.671947  4608 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:20.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.673190  4608 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap complete.
18:50:20.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.674929  4608 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent bootstrapping tablet: real 0.047s	user 0.038s	sys 0.007s
18:50:20.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.686357  4608 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:20.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.688210  4608 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Initialized, Role: FOLLOWER
18:50:20.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.689381  4608 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:20.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.691689  4608 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent starting tablet: real 0.017s	user 0.014s	sys 0.001s
18:50:20.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.700034  4581 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.66:42625
18:50:20.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.700063  4740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.66:42625 every 8 connection(s)
18:50:20.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.701551  4581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:20.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.706990  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 4581
18:50:20.707 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:20.727 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:20.730 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.67:39649
18:50:20.731 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:50:20.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.731571  4743 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:20.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.731622  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 4106
18:50:20.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.731976  4743 heartbeater.cc:464] Registering TS with master...
18:50:20.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.733053  4743 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:20.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.740255  3593 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:20.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.741137  4741 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:20.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.741356  4741 heartbeater.cc:464] Registering TS with master...
18:50:20.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.742035  4741 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:20.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.743853  3675 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.745571  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.66:38151
18:50:20.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.754168  4742 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:20.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.754382  4742 heartbeater.cc:464] Registering TS with master...
18:50:20.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.754927  4742 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:20.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.756513  3509 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:20.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.768946  4245 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:20.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:20.872408  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 4106
18:50:20.896 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:20.897 [DEBUG - kudu-nio-0] (Connection.java:239) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] handling channelInactive
18:50:20.897 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state READY due to: connection closed
18:50:20.897 [DEBUG - Time-limited test] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:20.899 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.899 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.899 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.922 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:20.923 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:20.923 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1596793039(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@72e3d38a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bbd4cec -> wakeup thread Time-limited test)) after lookup
18:50:20.924 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:20.925 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.925 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.926 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.942 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:20.942 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:20.943 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [29ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [29ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [44ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1596793039(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@72e3d38a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bbd4cec -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617))) after lookup
18:50:20.943 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:20.945 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.945 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.945 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.961 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:20.962 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:20.962 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=65), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [29ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [29ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [44ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [48ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [48ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [64ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1596793039(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@72e3d38a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bbd4cec -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925))) after lookup
18:50:20.963 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:20.964 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.965 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:20.965 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:21.083 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:21.083 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:21.084 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=187), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [29ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [29ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [44ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [48ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [48ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [64ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [66ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [68ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [68ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [182ms] refreshing cache from master, [182ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [185ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1596793039(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@72e3d38a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925) -> (continuation of Deferred@870850778 after retry RPC@1632622367), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bbd4cec -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925) -> (continuation of Deferred@870850778 after retry RPC@1632622367))) after lookup
18:50:21.085 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:21.087 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:21.087 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:21.087 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:21.582 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:21.582 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:21.583 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=685), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [29ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [29ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [44ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [48ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [48ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [64ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [66ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [68ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [68ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [182ms] refreshing cache from master, [182ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [185ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [188ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [190ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [191ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [681ms] refreshing cache from master, [682ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [684ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1596793039(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@72e3d38a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925) -> (continuation of Deferred@870850778 after retry RPC@1632622367) -> (continuation of Deferred@1525711730 after retry RPC@1167609884), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bbd4cec -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925) -> (continuation of Deferred@870850778 after retry RPC@1632622367) -> (continuation of Deferred@1525711730 after retry RPC@1167609884))) after lookup
18:50:21.584 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:21.585 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:21.585 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:21.585 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:21.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:21.748752  4741 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:22.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.102715  4758 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fa44f6302c4b4e11adf404ce13329e9d)
18:50:22.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.102929  4758 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:22.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.103549  4758 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:22.106297  4388 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:22.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.112339  4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b" is_pre_election: true
18:50:22.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.112816  4680 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 3.
18:50:22.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.113813  4387 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b; no voters: fa44f6302c4b4e11adf404ce13329e9d
18:50:22.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.114248  4758 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 4 FOLLOWER]: Leader pre-election won for term 5
18:50:22.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.114367  4758 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 4 FOLLOWER]: Starting leader election (detected failure of leader fa44f6302c4b4e11adf404ce13329e9d)
18:50:22.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.114440  4758 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 4 FOLLOWER]: Advancing to term 5
18:50:22.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.116529  4758 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:22.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.117172  4758 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 5 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:22.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.117686  4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:22.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.117931  4680 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 3 FOLLOWER]: Advancing to term 5
18:50:22.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:22.118726  4388 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:22.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.122207  4680 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 5.
18:50:22.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.122834  4387 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b; no voters: fa44f6302c4b4e11adf404ce13329e9d
18:50:22.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.123220  4758 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Leader election won for term 5
18:50:22.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.123718  4758 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 LEADER]: Becoming Leader. State: Replica: 32aab08028a947089da244a350b00c7e, State: Running, Role: LEADER
18:50:22.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.124296  4758 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:22.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.128594  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e reported cstate change: term changed from 4 to 5, leader changed from fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67) to 32aab08028a947089da244a350b00c7e (127.3.97.65). New cstate: current_term: 5 leader_uuid: "32aab08028a947089da244a350b00c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:22.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:22.562486  4388 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e -> Peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Couldn't send request to peer fa44f6302c4b4e11adf404ce13329e9d. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:22.662 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:22.663 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:22.663 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1766), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [29ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [29ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [44ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [48ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [48ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [61ms] refreshing cache from master, [61ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [64ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [66ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [68ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [68ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [182ms] refreshing cache from master, [182ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [185ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [188ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [190ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [191ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [681ms] refreshing cache from master, [682ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [684ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [687ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [688ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [688ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [1762ms] refreshing cache from master, [1762ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [1765ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1596793039(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@72e3d38a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925) -> (continuation of Deferred@870850778 after retry RPC@1632622367) -> (continuation of Deferred@1525711730 after retry RPC@1167609884) -> (continuation of Deferred@1075936633 after retry RPC@84369224), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@bbd4cec -> wakeup thread Time-limited test -> (continuation of Deferred@93013701 after retry RPC@1551131617) -> (continuation of Deferred@1027966663 after retry RPC@1112637925) -> (continuation of Deferred@870850778 after retry RPC@1632622367) -> (continuation of Deferred@1525711730 after retry RPC@1167609884) -> (continuation of Deferred@1075936633 after retry RPC@84369224))) after lookup
18:50:22.664 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:22.665 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] Successfully connected to peer
18:50:22.690 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x078f9f6a, L:/127.0.0.1:50834 - R:/127.3.97.65:42129] using TOKEN/null
18:50:22.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.695818  4680 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 5 FOLLOWER]: Refusing update from remote peer 32aab08028a947089da244a350b00c7e: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 14. (index mismatch)
18:50:22.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.696784  4779 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
18:50:22.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.717001  4778 mvcc.cc:205] Tried to move back new op lower bound from 7317398416152797184 to 7317398413825101824. Current Snapshot: MvccSnapshot[applied={T|T < 7317398416152797184}]
18:50:22.748 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.67:39649
18:50:22.749 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:22.749912  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:39649
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40339
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:23.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.119058  4789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:23.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.119402  4789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:23.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.119496  4789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:23.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.127573  4789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:23.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.127919  4789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.67
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.138279  4789 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:23.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:39649
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40339
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:23.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.139788  4789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:23.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.141637  4789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:23.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.156533  4796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:23.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.156488  4798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:23.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:23.157232  4795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:23.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.158062  4789 server_base.cc:1034] running on GCE node
18:50:23.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.158761  4789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:23.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.162931  4789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:23.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.166363  4789 hybrid_clock.cc:648] HybridClock initialized: now 1786474223166243 us; error 71 us; skew 500 ppm
18:50:23.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.166792  4789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:23.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.169587  4789 webserver.cc:460] Webserver started at http://127.3.97.67:40339/ using document root <none> and password file <none>
18:50:23.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.170429  4789 fs_manager.cc:359] Metadata directory not provided
18:50:23.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.170652  4789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:23.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.176076  4789 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.000s	sys 0.004s
18:50:23.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.179059  4806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:23.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.180325  4789 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
18:50:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.180537  4789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:11 on dist-test-slave-pgkr"
18:50:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.181546  4789 fs_report.cc:383] FS layout report
18:50:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:23.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/data
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:23.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:23.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.200379  4789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:23.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.201531  4789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:23.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.201926  4789 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:23.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.203146  4789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:23.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.205478  4816 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:23.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.212253  4789 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:23.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.212420  4789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
18:50:23.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.212567  4789 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.215804  4789 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:23.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.215927  4789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.000s	sys 0.002s
18:50:23.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.216272  4816 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap starting.
18:50:23.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.241858  4816 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Log is configured to *not* fsync() on all Append() calls
18:50:23.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.268019  4816 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:23.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.268934  4816 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap complete.
18:50:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.270560  4816 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent bootstrapping tablet: real 0.055s	user 0.039s	sys 0.012s
18:50:23.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.276957  4789 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.67:39649
18:50:23.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.277005  4937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.67:39649 every 8 connection(s)
18:50:23.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.278538  4789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:23.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.284477  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 4789
18:50:23.285 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:23.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.280491  4816 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.296422  4816 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa44f6302c4b4e11adf404ce13329e9d, State: Initialized, Role: FOLLOWER
18:50:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.297442  4816 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.305553  4816 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.004s
18:50:23.316 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:23.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.325947  4938 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:23.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.326325  4938 heartbeater.cc:464] Registering TS with master...
18:50:23.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.327361  4938 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:23.328 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.65:42129
18:50:23.329 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:23.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.329437  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 4367
18:50:23.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.330396  3675 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:23.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.331971  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.67:35049
18:50:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.333123  4940 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.333281  4940 heartbeater.cc:464] Registering TS with master...
18:50:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.333773  4940 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:23.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.335503  3593 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:23.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.335629  4938 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:23.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.338981  4939 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:23.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.339126  4939 heartbeater.cc:464] Registering TS with master...
18:50:23.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.339483  4939 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:23.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.340773  3509 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:23.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.348551  4515 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:23.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:23.450003  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 4367
18:50:23.475 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] handling channelInactive
18:50:23.475 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:23.475 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state READY due to: connection closed
18:50:23.476 [DEBUG - Time-limited test] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:23.478 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.478 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.478 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.501 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:23.502 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:23.502 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@138722758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3fb9b060 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48844362 -> wakeup thread Time-limited test)) after lookup
18:50:23.503 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:23.505 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.505 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.505 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:23.522 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:23.523 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [29ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [29ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@138722758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3fb9b060 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48844362 -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390))) after lookup
18:50:23.523 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:23.525 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.525 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.525 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.562 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:23.563 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:23.563 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=87), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [29ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [29ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [47ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [49ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [49ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [86ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@138722758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3fb9b060 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48844362 -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740))) after lookup
18:50:23.564 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:23.565 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.566 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.566 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.702 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:23.702 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:23.703 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=227), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [29ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [29ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [47ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [49ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [49ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [86ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [87ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [90ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [90ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [222ms] refreshing cache from master, [223ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [225ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@138722758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3fb9b060 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48844362 -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822))) after lookup
18:50:23.704 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:23.705 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.705 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:23.705 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:24.182 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:24.182 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:24.183 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=707), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [29ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [29ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [47ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [49ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [49ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [86ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [87ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [90ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [90ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [222ms] refreshing cache from master, [223ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [225ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [227ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [229ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [229ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [702ms] refreshing cache from master, [702ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [705ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@138722758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3fb9b060 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822) -> (continuation of Deferred@1535830170 after retry RPC@943913031), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48844362 -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822) -> (continuation of Deferred@1535830170 after retry RPC@943913031))) after lookup
18:50:24.184 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:24.185 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:24.185 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:24.186 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:24.782 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:24.783 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:24.784 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1307), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [29ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [29ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [47ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [49ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [49ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [86ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [87ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [90ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [90ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [222ms] refreshing cache from master, [223ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [225ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [227ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [229ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [229ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [702ms] refreshing cache from master, [702ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [705ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [707ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [710ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [710ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [1302ms] refreshing cache from master, [1302ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [1306ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@138722758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3fb9b060 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822) -> (continuation of Deferred@1535830170 after retry RPC@943913031) -> (continuation of Deferred@145485946 after retry RPC@654615016), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48844362 -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822) -> (continuation of Deferred@1535830170 after retry RPC@943913031) -> (continuation of Deferred@145485946 after retry RPC@654615016))) after lookup
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.784770  4954 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.785035  4954 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.786693  4954 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:24.795327  4603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:24.797628  4603 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.798121  4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.798684  4889 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 4.
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.799819  4603 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.800365  4954 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 5 FOLLOWER]: Leader pre-election won for term 6
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.800478  4954 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:24.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.800544  4954 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 5 FOLLOWER]: Advancing to term 6
18:50:24.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.802557  4954 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:24.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.803206  4954 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 6 election: Requested vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:24.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.804093  4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:24.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.804275  4889 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 4 FOLLOWER]: Advancing to term 6
18:50:24.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.808243  4889 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 6.
18:50:24.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.808780  4603 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 
18:50:24.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.809144  4954 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Leader election won for term 6
18:50:24.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.811650  4954 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 LEADER]: Becoming Leader. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Running, Role: LEADER
18:50:24.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:24.812871  4603 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:24.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.812721  4954 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:24.814 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:24.815 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:24.815 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:24.816 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:24.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:24.818997  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b reported cstate change: term changed from 5 to 6, leader changed from 32aab08028a947089da244a350b00c7e (127.3.97.65) to 857be9840834437fb311c99b57bf0b4b (127.3.97.66). New cstate: current_term: 6 leader_uuid: "857be9840834437fb311c99b57bf0b4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:25.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:25.273245  4889 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 6 FOLLOWER]: Refusing update from remote peer 857be9840834437fb311c99b57bf0b4b: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
18:50:25.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:25.274328  4954 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
18:50:25.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:25.295737  4603 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b -> Peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Couldn't send request to peer 32aab08028a947089da244a350b00c7e. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:27.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:27.950917  4603 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b -> Peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Couldn't send request to peer 32aab08028a947089da244a350b00c7e. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:50:28.662 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:28.663 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:28.663 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=5187), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [25ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [27ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [29ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [29ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [47ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [49ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [49ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [86ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [87ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [90ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [90ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [222ms] refreshing cache from master, [223ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [225ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [227ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [229ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [229ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [702ms] refreshing cache from master, [702ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [705ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [707ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [710ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [710ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [1302ms] refreshing cache from master, [1302ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [1306ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [1334ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [1339ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [1340ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [5183ms] refreshing cache from master, [5183ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [5186ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@138722758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@3fb9b060 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822) -> (continuation of Deferred@1535830170 after retry RPC@943913031) -> (continuation of Deferred@145485946 after retry RPC@654615016) -> (continuation of Deferred@1894346160 after retry RPC@1439455816), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48844362 -> wakeup thread Time-limited test -> (continuation of Deferred@1220204224 after retry RPC@583996390) -> (continuation of Deferred@1695603284 after retry RPC@1514460740) -> (continuation of Deferred@810606598 after retry RPC@1281029822) -> (continuation of Deferred@1535830170 after retry RPC@943913031) -> (continuation of Deferred@145485946 after retry RPC@654615016) -> (continuation of Deferred@1894346160 after retry RPC@1439455816))) after lookup
18:50:28.664 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:28.666 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] Successfully connected to peer
18:50:28.688 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xd3b385f8, L:/127.0.0.1:38406 - R:/127.3.97.66:42625] using TOKEN/null
18:50:28.719 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.65:42129
18:50:28.719 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:28.720160  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:28.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:29.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.089206  5012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:29.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.089545  5012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:29.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.089632  5012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:29.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.097720  5012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:29.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.097916  5012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.65
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.107717  5012 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:29.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:29.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.109246  5012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:29.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.111133  5012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:29.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.126451  5020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:29.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.126479  5019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:29.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.126521  5022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:29.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.127621  5012 server_base.cc:1034] running on GCE node
18:50:29.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.128304  5012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:29.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.129343  5012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:29.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.130685  5012 hybrid_clock.cc:648] HybridClock initialized: now 1786474229130519 us; error 64 us; skew 500 ppm
18:50:29.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.131096  5012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.138928  5012 webserver.cc:460] Webserver started at http://127.3.97.65:38971/ using document root <none> and password file <none>
18:50:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.139784  5012 fs_manager.cc:359] Metadata directory not provided
18:50:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.139914  5012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:29.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.145968  5012 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
18:50:29.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.149379  5029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:29.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.151126  5012 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.151293  5012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "32aab08028a947089da244a350b00c7e"
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.152346  5012 fs_report.cc:383] FS layout report
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/data
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:29.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:29.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:29.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.174947  5012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:29.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.175827  5012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:29.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.176088  5012 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:29.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.177289  5012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:29.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.179534  5038 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:29.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.186149  5012 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:29.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.186431  5012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
18:50:29.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.186576  5012 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:29.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.190246  5012 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:29.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.190378  5012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s	user 0.000s	sys 0.001s
18:50:29.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.190796  5038 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap starting.
18:50:29.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.219802  5038 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Log is configured to *not* fsync() on all Append() calls
18:50:29.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.236238  5012 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.65:42129
18:50:29.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.236291  5169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.65:42129 every 8 connection(s)
18:50:29.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.237535  5012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:29.247 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:29.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.246497  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 5012
18:50:29.260 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:79)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
18:50:29.261 [ERROR - Time-limited test] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203)
	io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296)
	org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431)
	org.apache.kudu.client.AsyncKuduScanner$AsyncKuduScannerBuilder.build(AsyncKuduScanner.java:1417)
	org.apache.kudu.client.TestMultipleLeaderFailover$1.get(TestMultipleLeaderFailover.java:48)
	org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
	org.apache.kudu.client.TestMultipleLeaderFailover.waitUntilRowCount(TestMultipleLeaderFailover.java:44)
	org.apache.kudu.client.TestMultipleLeaderFailover.testMultipleFailover(TestMultipleLeaderFailover.java:97)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	java.util.concurrent.FutureTask.run(FutureTask.java:266)
	java.lang.Thread.run(Thread.java:750)
18:50:29.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.281160  5038 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:29.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.281932  5038 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap complete.
18:50:29.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.283669  5038 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent bootstrapping tablet: real 0.093s	user 0.033s	sys 0.029s
18:50:29.304 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:29.305 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.66:42625
18:50:29.305 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:50:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.307787  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 4581
18:50:29.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.307125  5038 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:29.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.311853  5038 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aab08028a947089da244a350b00c7e, State: Initialized, Role: FOLLOWER
18:50:29.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.313946  5173 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:29.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.314311  5173 heartbeater.cc:464] Registering TS with master...
18:50:29.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.315107  5038 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:29.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.315485  5173 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:29.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.319555  3593 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:29.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.322789  5170 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:29.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.323019  5171 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:29.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.323195  5171 heartbeater.cc:464] Registering TS with master...
18:50:29.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.323272  5170 heartbeater.cc:464] Registering TS with master...
18:50:29.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.324048  4736 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:29.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.324263  5171 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:29.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.325568  5170 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:29.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.326323  3509 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:29.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.327879  5038 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent starting tablet: real 0.044s	user 0.012s	sys 0.009s
18:50:29.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.328204  3675 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:29.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.329888  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.65:53217
18:50:29.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.333062  5170 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:29.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:29.432648  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 4581
18:50:29.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:29.456 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] handling channelInactive
18:50:29.457 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state READY due to: connection closed
18:50:29.458 [DEBUG - Time-limited test] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:29.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:29.458276  5187 negotiation.cc:337] Failed RPC negotiation. Trace:
18:50:29.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.325248 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.97.66:44067 (local address 127.3.97.65:42129)
18:50:29.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.330097 (+  4849us) server_negotiation.cc:206] Beginning negotiation
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.330110 (+    13us) server_negotiation.cc:399] Waiting for connection header
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.330540 (+   430us) server_negotiation.cc:407] Connection header received
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.330725 (+   185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.330737 (+    12us) server_negotiation.cc:461] Received NEGOTIATE request from client
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.331134 (+   397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.331370 (+   236us) server_negotiation.cc:226] Negotiated authn=SASL
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.435770 (+104400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.437300 (+  1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:29.456792 (+ 19492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.97.66:44067: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:50:29.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":4618,"thread_start_us":6347,"threads_started":1}
18:50:29.459 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.460 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.460 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.481 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:29.482 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:29.482 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@171304996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@73b0ed93 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a98603c -> wakeup thread Time-limited test)) after lookup
18:50:29.483 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:29.485 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.485 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.485 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.502 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:29.503 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:29.503 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [26ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [28ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [29ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@171304996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@73b0ed93 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a98603c -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395))) after lookup
18:50:29.504 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:29.505 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.505 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.505 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:29.523 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:29.523 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=66), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [26ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [28ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [29ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [48ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [48ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@171304996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@73b0ed93 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a98603c -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118))) after lookup
18:50:29.524 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:29.525 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.525 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.526 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.562 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:29.562 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:29.563 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=105), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [26ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [28ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [29ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [48ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [48ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [66ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [69ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [69ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [101ms] refreshing cache from master, [102ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [104ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@171304996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@73b0ed93 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118) -> (continuation of Deferred@284480362 after retry RPC@1286555298), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a98603c -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118) -> (continuation of Deferred@284480362 after retry RPC@1286555298))) after lookup
18:50:29.564 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:29.565 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.566 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.566 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.583 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:29.583 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:29.584 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=126), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [26ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [28ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [29ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [48ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [48ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [66ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [69ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [69ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [101ms] refreshing cache from master, [102ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [104ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [107ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [109ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [109ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [122ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [125ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@171304996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@73b0ed93 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118) -> (continuation of Deferred@284480362 after retry RPC@1286555298) -> (continuation of Deferred@496226811 after retry RPC@1535276688), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a98603c -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118) -> (continuation of Deferred@284480362 after retry RPC@1286555298) -> (continuation of Deferred@496226811 after retry RPC@1535276688))) after lookup
18:50:29.585 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:29.591 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.592 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:29.592 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:30.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.525197  5188 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:30.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.525530  5188 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:30.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.527406  5188 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:30.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:30.529107  5031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:30.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.535089  4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:30.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:30.535533  5031 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:30.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.536170  5033 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32aab08028a947089da244a350b00c7e; no voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d
18:50:30.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.536608  5188 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
18:50:30.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.660521  5194 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 857be9840834437fb311c99b57bf0b4b)
18:50:30.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.660764  5194 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:30.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.662358  5194 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:30.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:30.663805  4810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:30.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:30.665601  4810 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:30.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.671191  5117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e" is_pre_election: true
18:50:30.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.671730  5117 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa44f6302c4b4e11adf404ce13329e9d in term 5.
18:50:30.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.672943  4811 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d; no voters: 857be9840834437fb311c99b57bf0b4b
18:50:30.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.673424  5194 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 6 FOLLOWER]: Leader pre-election won for term 7
18:50:30.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.673595  5194 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 857be9840834437fb311c99b57bf0b4b)
18:50:30.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.673720  5194 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 6 FOLLOWER]: Advancing to term 7
18:50:30.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.675879  5194 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:30.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.676595  5194 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 7 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:30.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.677310  5117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e"
18:50:30.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.677567  5117 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 5 FOLLOWER]: Advancing to term 7
18:50:30.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:30.678308  4810 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.682318  5117 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa44f6302c4b4e11adf404ce13329e9d in term 7.
18:50:30.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.682964  4811 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d; no voters: 857be9840834437fb311c99b57bf0b4b
18:50:30.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.683384  5194 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Leader election won for term 7
18:50:30.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.683884  5194 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 LEADER]: Becoming Leader. State: Replica: fa44f6302c4b4e11adf404ce13329e9d, State: Running, Role: LEADER
18:50:30.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.684433  5194 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:30.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:30.688822  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d reported cstate change: term changed from 6 to 7, leader changed from 857be9840834437fb311c99b57bf0b4b (127.3.97.66) to fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67). New cstate: current_term: 7 leader_uuid: "fa44f6302c4b4e11adf404ce13329e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: HEALTHY } } }
18:50:31.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:31.240880  5117 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 7 FOLLOWER]: Refusing update from remote peer fa44f6302c4b4e11adf404ce13329e9d: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
18:50:31.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:31.242007  5216 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
18:50:31.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:31.268813  4810 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d -> Peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Couldn't send request to peer 857be9840834437fb311c99b57bf0b4b. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:31.641 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:31.641 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:31.642 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2184), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [24ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [26ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [28ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [29ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [41ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [45ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [46ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [48ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [48ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [62ms] refreshing cache from master, [62ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [66ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [69ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [69ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [101ms] refreshing cache from master, [102ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [104ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [107ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [109ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [109ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [122ms] refreshing cache from master, [122ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [125ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [127ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [135ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [136ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [2181ms] refreshing cache from master, [2181ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [2184ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@171304996(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@73b0ed93 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118) -> (continuation of Deferred@284480362 after retry RPC@1286555298) -> (continuation of Deferred@496226811 after retry RPC@1535276688) -> (continuation of Deferred@136536915 after retry RPC@1079878193), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a98603c -> wakeup thread Time-limited test -> (continuation of Deferred@1562280812 after retry RPC@1418625395) -> (continuation of Deferred@1770033264 after retry RPC@1896194118) -> (continuation of Deferred@284480362 after retry RPC@1286555298) -> (continuation of Deferred@496226811 after retry RPC@1535276688) -> (continuation of Deferred@136536915 after retry RPC@1079878193))) after lookup
18:50:31.643 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:31.644 [DEBUG - kudu-nio-0] (Connection.java:757) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] Successfully connected to peer
18:50:31.665 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x16eae55e, L:/127.0.0.1:54978 - R:/127.3.97.67:39649] using TOKEN/null
18:50:31.696 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.66:42625
18:50:31.697 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:31.698505  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:42625
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36651
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:31.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:32.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.062219  5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:32.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.062546  5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:32.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.062705  5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:32.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.070500  5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:32.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.070791  5230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.66
18:50:32.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.080507  5230 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:32.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:32.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:32.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:42625
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36651
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:32.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:32.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.081920  5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:32.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.083856  5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:32.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.098367  5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:32.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.098341  5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:32.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:32.098897  5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:32.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.099918  5230 server_base.cc:1034] running on GCE node
18:50:32.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.100636  5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:32.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.101642  5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:32.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.102869  5230 hybrid_clock.cc:648] HybridClock initialized: now 1786474232102791 us; error 54 us; skew 500 ppm
18:50:32.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.103163  5230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:32.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.105635  5230 webserver.cc:460] Webserver started at http://127.3.97.66:36651/ using document root <none> and password file <none>
18:50:32.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.106314  5230 fs_manager.cc:359] Metadata directory not provided
18:50:32.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.106421  5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:32.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.112262  5230 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
18:50:32.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.115358  5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:32.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.116880  5230 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:50:32.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.117035  5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.118084  5230 fs_report.cc:383] FS layout report
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/data
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:32.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:32.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.139385  5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:32.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.140296  5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:32.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.140580  5230 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:32.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.141634  5230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:32.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.144052  5253 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:32.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.150172  5230 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:32.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.150329  5230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
18:50:32.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.150435  5230 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:32.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.154049  5230 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:32.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.154178  5230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s	user 0.001s	sys 0.000s
18:50:32.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.154561  5253 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap starting.
18:50:32.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.192960  5253 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Log is configured to *not* fsync() on all Append() calls
18:50:32.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.218451  5230 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.66:42625
18:50:32.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.218487  5384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.66:42625 every 8 connection(s)
18:50:32.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.220244  5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:32.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.222908  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 5230
18:50:32.225 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:32.253 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:32.254 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.67:39649
18:50:32.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:50:32.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.255333  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 4789
18:50:32.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.277792  5253 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:32.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.278910  5253 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap complete.
18:50:32.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.281183  5253 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent bootstrapping tablet: real 0.127s	user 0.063s	sys 0.012s
18:50:32.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.305469  4933 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:32.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.311113  5386 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:32.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.311455  5386 heartbeater.cc:464] Registering TS with master...
18:50:32.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.312443  5386 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:32.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.315222  3509 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:32.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.315896  5388 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:32.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.316047  5388 heartbeater.cc:464] Registering TS with master...
18:50:32.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.316529  5388 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:32.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.318389  5385 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:32.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.316350  5253 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:32.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.318531  5385 heartbeater.cc:464] Registering TS with master...
18:50:32.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.318557  5253 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Initialized, Role: FOLLOWER
18:50:32.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.319056  5385 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:32.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.319350  5253 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:32.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.321107  3675 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:32.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.322220  3593 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:32.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.322729  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.66:54669
18:50:32.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.322759  5253 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent starting tablet: real 0.041s	user 0.017s	sys 0.002s
18:50:32.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.325798  5385 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:32.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:32.418226  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 4789
18:50:32.443 [DEBUG - kudu-nio-0] (Connection.java:239) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] handling channelInactive
18:50:32.443 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state READY due to: connection closed
18:50:32.444 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:32.445 [DEBUG - Time-limited test] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:32.446 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.446 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.446 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.462 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:32.462 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:32.463 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1516621573(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@10a1d958 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@771760e6 -> wakeup thread Time-limited test)) after lookup
18:50:32.463 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:32.464 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.464 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.464 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.482 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:32.482 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:32.483 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [19ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [20ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1516621573(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@10a1d958 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@771760e6 -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838))) after lookup
18:50:32.483 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:32.485 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.485 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.485 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.502 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:32.502 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:32.502 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [19ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [20ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [39ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [41ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [54ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1516621573(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@10a1d958 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@771760e6 -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757))) after lookup
18:50:32.503 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:32.504 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.504 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.504 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.641 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:32.642 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:32.642 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=198), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [19ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [20ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [39ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [41ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [54ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [59ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [60ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [194ms] refreshing cache from master, [194ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [197ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1516621573(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@10a1d958 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757) -> (continuation of Deferred@427287835 after retry RPC@208156448), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@771760e6 -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757) -> (continuation of Deferred@427287835 after retry RPC@208156448))) after lookup
18:50:32.643 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:32.645 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.645 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.645 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.841 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:32.842 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:32.842 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=398), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [19ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [20ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [39ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [41ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [54ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [59ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [60ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [194ms] refreshing cache from master, [194ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [197ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [199ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [201ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [202ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [394ms] refreshing cache from master, [394ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [397ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1516621573(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@10a1d958 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757) -> (continuation of Deferred@427287835 after retry RPC@208156448) -> (continuation of Deferred@1249767397 after retry RPC@123801789), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@771760e6 -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757) -> (continuation of Deferred@427287835 after retry RPC@208156448) -> (continuation of Deferred@1249767397 after retry RPC@123801789))) after lookup
18:50:32.844 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:32.844 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.845 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:32.845 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:33.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.582489  5405 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:33.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.582841  5405 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:33.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.584494  5405 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:33.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:33.588255  5249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:33.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:33.589980  5249 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.592036  5117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e" is_pre_election: true
18:50:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.592875  5249 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b; no voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d
18:50:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.593329  5405 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
18:50:33.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.620389  5411 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fa44f6302c4b4e11adf404ce13329e9d)
18:50:33.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.620626  5411 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:33.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.621322  5411 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:33.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:33.623844  5033 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:33.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.630165  5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b" is_pre_election: true
18:50:33.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.630591  5328 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 6.
18:50:33.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.631664  5031 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b; no voters: fa44f6302c4b4e11adf404ce13329e9d
18:50:33.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.632009  5411 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 7 FOLLOWER]: Leader pre-election won for term 8
18:50:33.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.632125  5411 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 7 FOLLOWER]: Starting leader election (detected failure of leader fa44f6302c4b4e11adf404ce13329e9d)
18:50:33.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.632196  5411 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 7 FOLLOWER]: Advancing to term 8
18:50:33.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.634256  5411 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:33.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.634913  5411 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 8 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:33.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.635613  5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:33.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.635885  5328 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 6 FOLLOWER]: Advancing to term 8
18:50:33.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:33.636785  5033 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:33.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.640241  5328 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 8.
18:50:33.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.640725  5031 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b; no voters: fa44f6302c4b4e11adf404ce13329e9d
18:50:33.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.641041  5411 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 8 FOLLOWER]: Leader election won for term 8
18:50:33.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.641639  5411 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 8 LEADER]: Becoming Leader. State: Replica: 32aab08028a947089da244a350b00c7e, State: Running, Role: LEADER
18:50:33.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.642203  5411 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:33.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:33.645960  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e reported cstate change: term changed from 7 to 8, leader changed from fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67) to 32aab08028a947089da244a350b00c7e (127.3.97.65). New cstate: current_term: 8 leader_uuid: "32aab08028a947089da244a350b00c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:34.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:34.030691  5033 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e -> Peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Couldn't send request to peer fa44f6302c4b4e11adf404ce13329e9d. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:34.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:34.062060  5328 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 8 FOLLOWER]: Refusing update from remote peer 32aab08028a947089da244a350b00c7e: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
18:50:34.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:34.063060  5411 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
18:50:34.661 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:34.662 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:34.663 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2218), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [2ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [17ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [19ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [20ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [21ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [37ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [39ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [41ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [41ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [54ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [57ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [59ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [60ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [194ms] refreshing cache from master, [194ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [197ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [199ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [201ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [202ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [394ms] refreshing cache from master, [394ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [397ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [399ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [401ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [401ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [2214ms] refreshing cache from master, [2214ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [2217ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1516621573(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@10a1d958 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757) -> (continuation of Deferred@427287835 after retry RPC@208156448) -> (continuation of Deferred@1249767397 after retry RPC@123801789) -> (continuation of Deferred@1654796894 after retry RPC@197617802), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@771760e6 -> wakeup thread Time-limited test -> (continuation of Deferred@842319858 after retry RPC@1477245838) -> (continuation of Deferred@644283564 after retry RPC@428454757) -> (continuation of Deferred@427287835 after retry RPC@208156448) -> (continuation of Deferred@1249767397 after retry RPC@123801789) -> (continuation of Deferred@1654796894 after retry RPC@197617802))) after lookup
18:50:34.663 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:34.665 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] Successfully connected to peer
18:50:34.685 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xb4d03f2e, L:/127.0.0.1:51142 - R:/127.3.97.65:42129] using TOKEN/null
18:50:34.708 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.67:39649
18:50:34.708 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:34.709246  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:34.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:39649
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40339
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:35.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.059465  5447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:35.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.059830  5447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:35.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.059955  5447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:35.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.067893  5447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:35.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.068156  5447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.67
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.078008  5447 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:39649
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:35.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40339
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:35.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:35.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.080017  5447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:35.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.082716  5447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:35.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.097018  5454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:35.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.097091  5455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:35.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:35.097121  5457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:35.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.097967  5447 server_base.cc:1034] running on GCE node
18:50:35.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.098834  5447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:35.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.099957  5447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:35.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.101323  5447 hybrid_clock.cc:648] HybridClock initialized: now 1786474235101223 us; error 48 us; skew 500 ppm
18:50:35.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.101744  5447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:35.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.110111  5447 webserver.cc:460] Webserver started at http://127.3.97.67:40339/ using document root <none> and password file <none>
18:50:35.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.111143  5447 fs_manager.cc:359] Metadata directory not provided
18:50:35.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.111342  5447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:35.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.117913  5447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.006s
18:50:35.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.121811  5463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:35.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.123667  5447 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.002s
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.123934  5447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:11 on dist-test-slave-pgkr"
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.125224  5447 fs_report.cc:383] FS layout report
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/data
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:35.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:35.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:35.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.138782  5447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:35.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.140086  5447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:35.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.140506  5447 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:35.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.142014  5447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:35.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.144589  5471 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:35.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.151296  5447 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:35.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.151458  5447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
18:50:35.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.151592  5447 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:35.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.155138  5447 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:35.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.155267  5447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s	user 0.003s	sys 0.001s
18:50:35.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.155851  5471 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap starting.
18:50:35.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.193626  5471 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Log is configured to *not* fsync() on all Append() calls
18:50:35.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.230824  5447 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.67:39649
18:50:35.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.231221  5594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.67:39649 every 8 connection(s)
18:50:35.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.232662  5447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:35.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.235458  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 5447
18:50:35.236 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:35.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.237649  5471 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:35.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.238402  5471 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap complete.
18:50:35.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.240383  5471 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent bootstrapping tablet: real 0.085s	user 0.069s	sys 0.011s
18:50:35.266 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:35.267 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.65:42129
18:50:35.268 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.268896  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 5012
18:50:35.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.287210  5471 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:35.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.289273  5471 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa44f6302c4b4e11adf404ce13329e9d, State: Initialized, Role: FOLLOWER
18:50:35.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.290522  5471 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:35.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.301707  5595 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:35.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.302111  5595 heartbeater.cc:464] Registering TS with master...
18:50:35.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.303278  5595 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:35.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.306481  3676 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:35.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.307622  5597 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:35.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.307778  5597 heartbeater.cc:464] Registering TS with master...
18:50:35.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.308281  5597 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:35.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.308314  3676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.67:40427
18:50:35.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.308745  5471 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent starting tablet: real 0.068s	user 0.015s	sys 0.017s
18:50:35.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.309810  3592 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:35.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.311679  5165 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:35.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.312985  5595 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:35.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.314219  5596 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:35.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.314373  5596 heartbeater.cc:464] Registering TS with master...
18:50:35.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.314857  5596 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:35.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.316640  3509 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:35.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:35.414798  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 5012
18:50:35.439 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:35.439 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] handling channelInactive
18:50:35.440 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state READY due to: connection closed
18:50:35.440 [DEBUG - Time-limited test] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:35.441 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.442 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.442 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.461 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:35.461 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:35.462 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [21ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@860121861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@67d79697 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fa3d326 -> wakeup thread Time-limited test)) after lookup
18:50:35.463 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:35.465 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.465 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.465 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.482 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:35.482 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:35.482 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [21ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [22ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [25ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [25ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@860121861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@67d79697 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fa3d326 -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779))) after lookup
18:50:35.483 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:35.484 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.484 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.484 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.522 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:35.523 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:35.523 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [21ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [22ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [25ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [25ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [43ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@860121861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@67d79697 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fa3d326 -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934))) after lookup
18:50:35.524 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:35.526 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.526 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.526 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.622 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:35.622 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:35.623 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=183), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [21ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [22ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [25ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [25ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [43ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [84ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [86ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [86ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [182ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@860121861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@67d79697 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fa3d326 -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211))) after lookup
18:50:35.624 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:35.625 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.625 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.625 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.701 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:35.702 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:35.702 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=262), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [21ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [22ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [25ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [25ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [43ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [84ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [86ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [86ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [182ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [183ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [185ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [185ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [258ms] refreshing cache from master, [258ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [261ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@860121861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@67d79697 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211) -> (continuation of Deferred@1722796176 after retry RPC@1669068395), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fa3d326 -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211) -> (continuation of Deferred@1722796176 after retry RPC@1669068395))) after lookup
18:50:35.703 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:35.705 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.705 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:35.705 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:36.381 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:36.382 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:36.382 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=942), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [21ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [22ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [25ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [25ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [43ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [84ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [86ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [86ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [182ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [183ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [185ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [185ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [258ms] refreshing cache from master, [258ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [261ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [263ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [265ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [265ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [938ms] refreshing cache from master, [939ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [941ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@860121861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@67d79697 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211) -> (continuation of Deferred@1722796176 after retry RPC@1669068395) -> (continuation of Deferred@1443997820 after retry RPC@1793442939), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fa3d326 -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211) -> (continuation of Deferred@1722796176 after retry RPC@1669068395) -> (continuation of Deferred@1443997820 after retry RPC@1793442939))) after lookup
18:50:36.383 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:36.384 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:36.384 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:36.384 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:36.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.464349  5611 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:36.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.464731  5611 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:36.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.466777  5611 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:36.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:36.468719  5466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:36.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:36.470338  5466 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:36.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.474419  5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b" is_pre_election: true
18:50:36.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.475351  5465 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b
18:50:36.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.475733  5611 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
18:50:36.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.784365  5618 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:36.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.784649  5618 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:36.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.785418  5618 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:36.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:36.788822  5249 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:36.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.794526  5544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:36.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.795112  5544 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 7.
18:50:36.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.796221  5249 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:36.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.796593  5618 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 8 FOLLOWER]: Leader pre-election won for term 9
18:50:36.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.796723  5618 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:36.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.796859  5618 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 8 FOLLOWER]: Advancing to term 9
18:50:36.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.799198  5618 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:36.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.799940  5618 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 9 election: Requested vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:36.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.800827  5544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:36.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.801052  5544 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 7 FOLLOWER]: Advancing to term 9
18:50:36.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:36.801385  5249 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:36.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.805351  5544 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 9.
18:50:36.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.805891  5249 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:36.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.806201  5618 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Leader election won for term 9
18:50:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.806761  5618 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 LEADER]: Becoming Leader. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Running, Role: LEADER
18:50:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.807291  5618 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:36.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:36.810952  3676 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b reported cstate change: term changed from 8 to 9, leader changed from 32aab08028a947089da244a350b00c7e (127.3.97.65) to 857be9840834437fb311c99b57bf0b4b (127.3.97.66). New cstate: current_term: 9 leader_uuid: "857be9840834437fb311c99b57bf0b4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:37.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:37.255662  5249 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b -> Peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Couldn't send request to peer 32aab08028a947089da244a350b00c7e. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:37.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:37.354297  5544 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 9 FOLLOWER]: Refusing update from remote peer 857be9840834437fb311c99b57bf0b4b: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
18:50:37.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:37.355384  5618 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
18:50:38.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:38.772123  5249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111) [suppressed 5 similar messages]
18:50:39.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:39.628621  5249 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b -> Peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Couldn't send request to peer 32aab08028a947089da244a350b00c7e. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
18:50:39.881 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:39.882 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:39.882 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4442), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [2ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [21ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [22ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [25ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [25ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [41ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [43ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [44ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [44ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [78ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [81ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [84ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [86ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [86ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [182ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [183ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [185ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [185ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [258ms] refreshing cache from master, [258ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [261ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [263ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [265ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [265ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [938ms] refreshing cache from master, [939ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [941ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [942ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [944ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [944ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [4438ms] refreshing cache from master, [4438ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [4441ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@860121861(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@67d79697 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211) -> (continuation of Deferred@1722796176 after retry RPC@1669068395) -> (continuation of Deferred@1443997820 after retry RPC@1793442939) -> (continuation of Deferred@1456571428 after retry RPC@1111958317), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fa3d326 -> wakeup thread Time-limited test -> (continuation of Deferred@17763943 after retry RPC@1089787779) -> (continuation of Deferred@890644876 after retry RPC@1459040934) -> (continuation of Deferred@1088730771 after retry RPC@795463211) -> (continuation of Deferred@1722796176 after retry RPC@1669068395) -> (continuation of Deferred@1443997820 after retry RPC@1793442939) -> (continuation of Deferred@1456571428 after retry RPC@1111958317))) after lookup
18:50:39.883 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:39.885 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] Successfully connected to peer
18:50:39.904 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xe75796d0, L:/127.0.0.1:50386 - R:/127.3.97.66:42625] using TOKEN/null
18:50:39.929 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.65:42129
18:50:39.929 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:39.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:39.930576  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:39.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:39.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:39.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:39.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:40.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.300266  5663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:40.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.300678  5663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:40.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.300806  5663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:40.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.308375  5663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:40.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.308636  5663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.65
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.318145  5663 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:40.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:40.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:40.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.319937  5663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:40.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.321764  5663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:40.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.336440  5669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:40.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.336472  5668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:40.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.336395  5671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:40.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.337568  5663 server_base.cc:1034] running on GCE node
18:50:40.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.338177  5663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:40.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.339239  5663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:40.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.340494  5663 hybrid_clock.cc:648] HybridClock initialized: now 1786474240340403 us; error 56 us; skew 500 ppm
18:50:40.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.340847  5663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:40.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.348635  5663 webserver.cc:460] Webserver started at http://127.3.97.65:38971/ using document root <none> and password file <none>
18:50:40.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.349527  5663 fs_manager.cc:359] Metadata directory not provided
18:50:40.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.349696  5663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.355552  5663 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
18:50:40.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.359105  5678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:40.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.360533  5663 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
18:50:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.360736  5663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "32aab08028a947089da244a350b00c7e"
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.361790  5663 fs_report.cc:383] FS layout report
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/data
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:40.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:40.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:40.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:40.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.377266  5663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:40.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.378403  5663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:40.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.378772  5663 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:40.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.382479  5663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.385000  5685 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.390990  5663 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.391148  5663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
18:50:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.391319  5663 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:40.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.394641  5663 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:40.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.394755  5663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.000s	sys 0.004s
18:50:40.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.395180  5685 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap starting.
18:50:40.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.429558  5685 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Log is configured to *not* fsync() on all Append() calls
18:50:40.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.455139  5663 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.65:42129
18:50:40.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.455233  5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.65:42129 every 8 connection(s)
18:50:40.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.456864  5663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:40.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.466234  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 5663
18:50:40.466 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:40.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.480409  5685 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:40.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.481235  5685 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap complete.
18:50:40.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.483060  5685 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent bootstrapping tablet: real 0.088s	user 0.054s	sys 0.030s
18:50:40.485 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:40.488 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.66:42625
18:50:40.488 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:50:40.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.488977  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 5230
18:50:40.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.493258  5685 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:40.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.495071  5685 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aab08028a947089da244a350b00c7e, State: Initialized, Role: FOLLOWER
18:50:40.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.495920  5685 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:40.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.519869  5807 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:40.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.520264  5807 heartbeater.cc:464] Registering TS with master...
18:50:40.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.521401  5807 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:40.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.522434  5685 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent starting tablet: real 0.039s	user 0.017s	sys 0.020s
18:50:40.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.524597  3675 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:40.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.526170  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.65:48749
18:50:40.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.527256  5380 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:40.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.530264  5807 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:40.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.531327  5809 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:40.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.531487  5809 heartbeater.cc:464] Registering TS with master...
18:50:40.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.531879  5809 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:40.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.533404  3592 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:40.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.536918  5808 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:40.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.537042  5808 heartbeater.cc:464] Registering TS with master...
18:50:40.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.537322  5808 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:40.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.538579  3509 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:40.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:40.634864  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 5230
18:50:40.662 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:40.662 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] handling channelInactive
18:50:40.662 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state READY due to: connection closed
18:50:40.663 [DEBUG - Time-limited test] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:40.663698  5817 negotiation.cc:337] Failed RPC negotiation. Trace:
18:50:40.664 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.475076 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.97.66:38527 (local address 127.3.97.65:42129)
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.479493 (+  4417us) server_negotiation.cc:206] Beginning negotiation
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.479508 (+    15us) server_negotiation.cc:399] Waiting for connection header
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.479564 (+    56us) server_negotiation.cc:407] Connection header received
18:50:40.665 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.482344 (+  2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.482380 (+    36us) server_negotiation.cc:461] Received NEGOTIATE request from client
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.505554 (+ 23174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.505802 (+   248us) server_negotiation.cc:226] Negotiated authn=SASL
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.510568 (+  4766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.516026 (+  5458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.528235 (+ 12209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.529062 (+   827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
18:50:40.665 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.531101 (+  2039us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:40.662150 (+131049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.97.66:38527: BlockingRecv error: recv got EOF from 127.3.97.66:38527 (error 108)
18:50:40.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":3883,"thread_start_us":3715,"threads_started":1}
18:50:40.682 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:40.682 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:40.683 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [19ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@2004929649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@4973ea62 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f966315 -> wakeup thread Time-limited test)) after lookup
18:50:40.683 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:40.685 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.685 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.686 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.702 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:40.702 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:40.702 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [19ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [21ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [23ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [24ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [39ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@2004929649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@4973ea62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f966315 -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068))) after lookup
18:50:40.703 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:40.704 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.704 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.704 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.721 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:40.721 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:40.722 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [19ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [21ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [23ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [24ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [39ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [42ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [42ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [59ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@2004929649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@4973ea62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f966315 -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053))) after lookup
18:50:40.722 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:40.724 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.724 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.724 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.842 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:40.842 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:40.842 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=180), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [19ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [21ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [23ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [24ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [39ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [42ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [42ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [59ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [60ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [62ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [62ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [179ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@2004929649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@4973ea62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@577710931 after retry RPC@1445241616), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f966315 -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@577710931 after retry RPC@1445241616))) after lookup
18:50:40.843 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:40.844 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.844 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:40.844 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:41.322 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:41.323 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:41.323 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=661), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [19ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [21ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [23ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [24ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [39ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [42ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [42ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [59ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [60ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [62ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [62ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [179ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [181ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [182ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [182ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [657ms] refreshing cache from master, [657ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [660ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@2004929649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@4973ea62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@577710931 after retry RPC@1445241616) -> (continuation of Deferred@1984812233 after retry RPC@654619180), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f966315 -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@577710931 after retry RPC@1445241616) -> (continuation of Deferred@1984812233 after retry RPC@654619180))) after lookup
18:50:41.324 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:41.326 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:41.326 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state CONNECTING due to: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:41.326 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625
18:50:41.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.862195  5825 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 857be9840834437fb311c99b57bf0b4b)
18:50:41.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.862399  5825 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:41.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.863122  5825 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:41.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:41.864331  5465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:41.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:41.865948  5465 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:41.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.872442  5756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e" is_pre_election: true
18:50:41.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.872906  5756 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa44f6302c4b4e11adf404ce13329e9d in term 8.
18:50:41.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.873873  5466 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d; no voters: 857be9840834437fb311c99b57bf0b4b
18:50:41.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.874199  5825 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 9 FOLLOWER]: Leader pre-election won for term 10
18:50:41.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.874329  5825 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 857be9840834437fb311c99b57bf0b4b)
18:50:41.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.874403  5825 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 9 FOLLOWER]: Advancing to term 10
18:50:41.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.876524  5825 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:41.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.877142  5825 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 10 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:41.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.877673  5756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e"
18:50:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.877912  5756 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 8 FOLLOWER]: Advancing to term 10
18:50:41.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:41.878702  5465 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111)
18:50:41.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.882150  5756 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa44f6302c4b4e11adf404ce13329e9d in term 10.
18:50:41.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.882695  5466 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d; no voters: 857be9840834437fb311c99b57bf0b4b
18:50:41.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.882982  5825 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Leader election won for term 10
18:50:41.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.883456  5825 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 LEADER]: Becoming Leader. State: Replica: fa44f6302c4b4e11adf404ce13329e9d, State: Running, Role: LEADER
18:50:41.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.883980  5825 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:41.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:41.887552  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d reported cstate change: term changed from 9 to 10, leader changed from 857be9840834437fb311c99b57bf0b4b (127.3.97.66) to fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67). New cstate: current_term: 10 leader_uuid: "fa44f6302c4b4e11adf404ce13329e9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: HEALTHY } } }
18:50:42.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:42.282439  5756 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 10 FOLLOWER]: Refusing update from remote peer fa44f6302c4b4e11adf404ce13329e9d: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
18:50:42.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:42.283427  5825 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
18:50:42.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:42.299139  5465 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d -> Peer 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625): Couldn't send request to peer 857be9840834437fb311c99b57bf0b4b. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.66:42625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:43.282 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:43.282 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:43.282 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2620), Traces: [0ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [3ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [3ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [19ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [21ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [23ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [24ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [39ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [41ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [42ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [42ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [59ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [60ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [62ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [62ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [177ms] refreshing cache from master, [177ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [179ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [181ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [182ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [182ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [657ms] refreshing cache from master, [657ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [660ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [662ms] sending RPC to server 857be9840834437fb311c99b57bf0b4b, [664ms] received response from server 857be9840834437fb311c99b57bf0b4b: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [664ms] delaying RPC due to: Network error: Failed to connect to peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625): Connection refused: /127.3.97.66:42625, [2616ms] refreshing cache from master, [2617ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [2619ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@2004929649(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@4973ea62 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@577710931 after retry RPC@1445241616) -> (continuation of Deferred@1984812233 after retry RPC@654619180) -> (continuation of Deferred@884277175 after retry RPC@908312287), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7f966315 -> wakeup thread Time-limited test -> (continuation of Deferred@1023459194 after retry RPC@1908600068) -> (continuation of Deferred@1860270832 after retry RPC@1114037053) -> (continuation of Deferred@577710931 after retry RPC@1445241616) -> (continuation of Deferred@1984812233 after retry RPC@654619180) -> (continuation of Deferred@884277175 after retry RPC@908312287))) after lookup
18:50:43.283 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:43.284 [DEBUG - kudu-nio-0] (Connection.java:757) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] Successfully connected to peer
18:50:43.305 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x950ea227, L:/127.0.0.1:60174 - R:/127.3.97.67:39649] using TOKEN/null
18:50:43.329 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.66:42625
18:50:43.329 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.330446  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:43.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:42625
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36651
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:43.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.694420  5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:43.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.694804  5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:43.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.694898  5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:43.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.703347  5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:43.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.703523  5867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.66
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.712885  5867 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.66:42625
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.66
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36651
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.66
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/logs
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:43.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:43.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:43.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:43.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:43.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:43.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.714227  5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:43.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.716013  5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:43.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.729667  5873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:43.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.729705  5872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:43.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:43.729912  5875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:43.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.730521  5867 server_base.cc:1034] running on GCE node
18:50:43.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.731202  5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:43.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.732191  5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:43.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.733464  5867 hybrid_clock.cc:648] HybridClock initialized: now 1786474243733369 us; error 53 us; skew 500 ppm
18:50:43.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.733740  5867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.736076  5867 webserver.cc:460] Webserver started at http://127.3.97.66:36651/ using document root <none> and password file <none>
18:50:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.736792  5867 fs_manager.cc:359] Metadata directory not provided
18:50:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.736948  5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:43.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.742180  5867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.001s
18:50:43.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.745241  5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:43.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.746723  5867 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
18:50:43.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.746905  5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.747938  5867 fs_report.cc:383] FS layout report
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/wal
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/data
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:43.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.777207  5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:43.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.778227  5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:43.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.778597  5867 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:43.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.779817  5867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:43.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.781951  5891 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:43.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.787708  5867 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:43.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.787889  5867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
18:50:43.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.788028  5867 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.791453  5867 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:43.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.791584  5867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s	user 0.003s	sys 0.000s
18:50:43.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.791909  5891 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap starting.
18:50:43.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.822732  5891 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Log is configured to *not* fsync() on all Append() calls
18:50:43.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.837723  5867 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.66:42625
18:50:43.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.837759  6018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.66:42625 every 8 connection(s)
18:50:43.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.839192  5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-1/data/info.pb
18:50:43.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.844514  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 5867
18:50:43.845 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:43.885 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:43.886 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.67:39649
18:50:43.886 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:50:43.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.887197  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 5447
18:50:43.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.902721  5589 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:43.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.903054  6019 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:43.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.903415  6019 heartbeater.cc:464] Registering TS with master...
18:50:43.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.904409  6021 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:43.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.904549  6021 heartbeater.cc:464] Registering TS with master...
18:50:43.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.905180  5891 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:43.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.905922  5891 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Bootstrap complete.
18:50:43.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.907358  6019 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:43.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.907794  5891 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent bootstrapping tablet: real 0.116s	user 0.060s	sys 0.024s
18:50:43.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.908500  6021 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:43.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.909653  6022 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:43.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.909819  6022 heartbeater.cc:464] Registering TS with master...
18:50:43.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.910312  6022 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:43.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.911132  3675 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:43.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.912039  3592 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:43.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.912369  3509 ts_manager.cc:194] Re-registered known tserver with Master: 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625)
18:50:43.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.912619  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.66:43841
18:50:43.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.916038  5891 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:43.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.917009  5891 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Initialized, Role: FOLLOWER
18:50:43.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.917721  5891 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:43.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.924854  6019 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:43.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:43.925526  5891 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b: Time spent starting tablet: real 0.018s	user 0.011s	sys 0.006s
18:50:44.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:44.013129  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 5447
18:50:44.038 [DEBUG - kudu-nio-0] (Connection.java:239) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] handling channelInactive
18:50:44.038 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:44.038 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state READY due to: connection closed
18:50:44.039 [DEBUG - Time-limited test] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:44.040 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.041 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.041 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.061 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:44.062 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:44.062 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [3ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1466451300(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@105bd761 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38cdc79c -> wakeup thread Time-limited test)) after lookup
18:50:44.063 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:44.064 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.064 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.064 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.081 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:44.081 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:44.081 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [3ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [26ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [26ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1466451300(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@105bd761 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38cdc79c -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858))) after lookup
18:50:44.082 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:44.083 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.083 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.083 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.101 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:44.101 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:44.101 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=63), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [3ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [26ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [26ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [44ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [45ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [46ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [62ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1466451300(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@105bd761 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38cdc79c -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047))) after lookup
18:50:44.102 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:44.103 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.103 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.103 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.241 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:44.242 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:44.244 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=205), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [3ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [26ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [26ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [44ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [45ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [46ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [62ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [64ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [65ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [65ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [200ms] refreshing cache from master, [200ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [203ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1466451300(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@105bd761 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38cdc79c -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712))) after lookup
18:50:44.245 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:44.247 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.247 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.247 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.542 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:44.542 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:44.543 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=504), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [3ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [26ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [26ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [44ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [45ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [46ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [62ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [64ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [65ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [65ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [200ms] refreshing cache from master, [200ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [203ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [207ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [209ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [209ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [500ms] refreshing cache from master, [501ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [503ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1466451300(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@105bd761 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712) -> (continuation of Deferred@1968413044 after retry RPC@1636604339), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38cdc79c -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712) -> (continuation of Deferred@1968413044 after retry RPC@1636604339))) after lookup
18:50:44.543 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:44.544 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.544 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:44.544 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:45.082 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:45.082 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)[L]]}]
18:50:45.083 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1044), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [3ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [26ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [26ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [44ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [45ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [46ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [62ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [64ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [65ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [65ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [200ms] refreshing cache from master, [200ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [203ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [207ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [209ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [209ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [500ms] refreshing cache from master, [501ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [503ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [505ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [506ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [506ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [1040ms] refreshing cache from master, [1040ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [1043ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1466451300(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@105bd761 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712) -> (continuation of Deferred@1968413044 after retry RPC@1636604339) -> (continuation of Deferred@823149982 after retry RPC@291752754), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38cdc79c -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712) -> (continuation of Deferred@1968413044 after retry RPC@1636604339) -> (continuation of Deferred@823149982 after retry RPC@291752754))) after lookup
18:50:45.083 [DEBUG - kudu-nio-0] (Connection.java:748) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] connecting to peer
18:50:45.084 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:45.084 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] cleaning up while in state CONNECTING due to: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:45.084 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649
18:50:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.291780  6036 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.292027  6036 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:45.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.293603  6036 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:45.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:45.296100  5884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:45.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:45.298362  5884 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:45.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.301512  5756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "32aab08028a947089da244a350b00c7e" is_pre_election: true
18:50:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.302273  5884 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b; no voters: 32aab08028a947089da244a350b00c7e, fa44f6302c4b4e11adf404ce13329e9d
18:50:45.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.302709  6036 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
18:50:45.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.402889  6042 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader fa44f6302c4b4e11adf404ce13329e9d)
18:50:45.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.403151  6042 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:45.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.404714  6042 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:45.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:45.406558  5681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:45.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:45.408147  5681 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:45.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.413134  5967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b" is_pre_election: true
18:50:45.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.413538  5967 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 9.
18:50:45.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.414533  5680 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b; no voters: fa44f6302c4b4e11adf404ce13329e9d
18:50:45.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.415019  6042 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 10 FOLLOWER]: Leader pre-election won for term 11
18:50:45.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.415160  6042 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 10 FOLLOWER]: Starting leader election (detected failure of leader fa44f6302c4b4e11adf404ce13329e9d)
18:50:45.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.415288  6042 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 10 FOLLOWER]: Advancing to term 11
18:50:45.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.417367  6042 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:45.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.418047  6042 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 11 election: Requested vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:45.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.418563  5967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "32aab08028a947089da244a350b00c7e" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b"
18:50:45.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.418828  5967 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 9 FOLLOWER]: Advancing to term 11
18:50:45.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:45.419737  5681 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111)
18:50:45.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.423038  5967 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32aab08028a947089da244a350b00c7e in term 11.
18:50:45.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.423475  5680 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b; no voters: fa44f6302c4b4e11adf404ce13329e9d
18:50:45.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.423779  6042 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 11 FOLLOWER]: Leader election won for term 11
18:50:45.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.424273  6042 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 11 LEADER]: Becoming Leader. State: Replica: 32aab08028a947089da244a350b00c7e, State: Running, Role: LEADER
18:50:45.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.424774  6042 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:45.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.428947  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e reported cstate change: term changed from 10 to 11, leader changed from fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67) to 32aab08028a947089da244a350b00c7e (127.3.97.65). New cstate: current_term: 11 leader_uuid: "32aab08028a947089da244a350b00c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:45.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.996253  5967 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 11 FOLLOWER]: Refusing update from remote peer 32aab08028a947089da244a350b00c7e: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
18:50:45.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:45.997241  6062 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
18:50:45.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:45.999207  5681 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e -> Peer fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649): Couldn't send request to peer fa44f6302c4b4e11adf404ce13329e9d. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.67:39649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:46.102 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:46.102 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:46.103 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2064), Traces: [0ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [3ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [3ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [20ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [23ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [26ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [26ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [43ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [44ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [45ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [46ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [60ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [62ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [64ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [65ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [65ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [200ms] refreshing cache from master, [200ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [203ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [207ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [209ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [209ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [500ms] refreshing cache from master, [501ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [503ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [505ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [506ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [506ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [1040ms] refreshing cache from master, [1040ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [1043ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [1045ms] sending RPC to server fa44f6302c4b4e11adf404ce13329e9d, [1046ms] received response from server fa44f6302c4b4e11adf404ce13329e9d: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [1046ms] delaying RPC due to: Network error: Failed to connect to peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649): Connection refused: /127.3.97.67:39649, [2060ms] refreshing cache from master, [2060ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [2063ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@1466451300(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@105bd761 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712) -> (continuation of Deferred@1968413044 after retry RPC@1636604339) -> (continuation of Deferred@823149982 after retry RPC@291752754) -> (continuation of Deferred@1549624107 after retry RPC@1407715785), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@38cdc79c -> wakeup thread Time-limited test -> (continuation of Deferred@92749416 after retry RPC@690795858) -> (continuation of Deferred@2110108048 after retry RPC@823070047) -> (continuation of Deferred@1211272002 after retry RPC@73532712) -> (continuation of Deferred@1968413044 after retry RPC@1636604339) -> (continuation of Deferred@823149982 after retry RPC@291752754) -> (continuation of Deferred@1549624107 after retry RPC@1407715785))) after lookup
18:50:46.104 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:46.104 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] Successfully connected to peer
18:50:46.126 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x742f88fd, L:/127.0.0.1:52640 - R:/127.3.97.65:42129] using TOKEN/null
18:50:46.149 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.67:39649
18:50:46.150 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.150990  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:39649
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40339
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:46.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:46.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:46.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.628229  6080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:46.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.629195  6080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:46.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.629462  6080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:46.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.657050  6080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:46.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.657683  6080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.67
18:50:46.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.687129  6080 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:46.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.67:39649
18:50:46.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:46.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.67
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40339
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:46.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.67
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/logs
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:46.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:46.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.690711  6080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:46.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.696599  6080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:46.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.745520  6087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:46.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.746074  6086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:46.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:46.748121  6089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:46.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.748637  6080 server_base.cc:1034] running on GCE node
18:50:46.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.751080  6080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:46.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.754998  6080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:46.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.759019  6080 hybrid_clock.cc:648] HybridClock initialized: now 1786474246758693 us; error 261 us; skew 500 ppm
18:50:46.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.760078  6080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.774019  6080 webserver.cc:460] Webserver started at http://127.3.97.67:40339/ using document root <none> and password file <none>
18:50:46.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.776691  6080 fs_manager.cc:359] Metadata directory not provided
18:50:46.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.777034  6080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:46.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.797266  6080 fs_manager.cc:709] Time spent opening directory manager: real 0.013s	user 0.010s	sys 0.001s
18:50:46.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.808421  6095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:46.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.813683  6080 fs_manager.cc:725] Time spent opening block manager: real 0.010s	user 0.002s	sys 0.003s
18:50:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.814255  6080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:11 on dist-test-slave-pgkr"
18:50:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.817463  6080 fs_report.cc:383] FS layout report
18:50:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/wal
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/data
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:46.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:46.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:46.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:46.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:46.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.884060  6080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:46.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.888337  6080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:46.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.889981  6080 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:46.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.895681  6080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:46.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.907496  6102 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.936154  6080 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:46.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.936980  6080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.032s	user 0.000s	sys 0.004s
18:50:46.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.937980  6080 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:46.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.956359  6080 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:46.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.956897  6080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.019s	user 0.020s	sys 0.000s
18:50:46.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:46.957567  6102 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap starting.
18:50:47.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.154940  6102 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Log is configured to *not* fsync() on all Append() calls
18:50:47.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.235435  6080 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.67:39649
18:50:47.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.235703  6229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.67:39649 every 8 connection(s)
18:50:47.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.241881  6080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-2/data/info.pb
18:50:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.253687  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 6080
18:50:47.257 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:47.388 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:47.390 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.3.97.65:42129
18:50:47.392 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:47.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.393388  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 5663
18:50:47.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.440021  6232 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:47.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.441715  6232 heartbeater.cc:464] Registering TS with master...
18:50:47.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.446312  6231 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:47.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.447031  6231 heartbeater.cc:464] Registering TS with master...
18:50:47.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.449155  6231 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:47.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.463006  6232 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:47.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.470223  5800 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:47.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.482069  3508 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:47.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.493201  3675 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:47.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.496587  6235 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:47.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.497184  6235 heartbeater.cc:464] Registering TS with master...
18:50:47.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.499562  6235 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:47.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.501119  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.67:44379
18:50:47.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.509419  3592 ts_manager.cc:194] Re-registered known tserver with Master: fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:47.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.548045  6102 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:47.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.550405  6102 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Bootstrap complete.
18:50:47.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.555411  6102 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent bootstrapping tablet: real 0.599s	user 0.334s	sys 0.091s
18:50:47.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.577855  6102 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:47.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.581743  6102 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa44f6302c4b4e11adf404ce13329e9d, State: Initialized, Role: FOLLOWER
18:50:47.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.585204  6102 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:47.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.590206  6231 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:47.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.593665  6102 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d: Time spent starting tablet: real 0.038s	user 0.032s	sys 0.004s
18:50:47.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:47.795105  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 5663
18:50:47.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:47.881510  5883 connection.cc:535] server connection from 127.3.97.65:37825 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:50:47.883 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] handling channelInactive
18:50:47.884 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state READY due to: connection closed
18:50:47.886 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:47.889 [DEBUG - Time-limited test] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:47.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:47.890091  6245 negotiation.cc:337] Failed RPC negotiation. Trace:
18:50:47.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.450569 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.97.65:35141 (local address 127.3.97.67:39649)
18:50:47.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.486187 (+ 35618us) server_negotiation.cc:206] Beginning negotiation
18:50:47.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.486207 (+    20us) server_negotiation.cc:399] Waiting for connection header
18:50:47.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.486365 (+   158us) server_negotiation.cc:407] Connection header received
18:50:47.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.487069 (+   704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
18:50:47.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.487122 (+    53us) server_negotiation.cc:461] Received NEGOTIATE request from client
18:50:47.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.488805 (+  1683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
18:50:47.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.490971 (+  2166us) server_negotiation.cc:226] Negotiated authn=SASL
18:50:47.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.797464 (+306493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
18:50:47.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.802565 (+  5101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
18:50:47.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0811 18:50:47.882693 (+ 80128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.97.65:35141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
18:50:47.893 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":34613,"thread_start_us":36874,"threads_started":1}
18:50:47.893 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.894 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.908 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:47.909 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:47.910 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [6ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [7ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@393441439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@13b6822e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b14ce23 -> wakeup thread Time-limited test)) after lookup
18:50:47.912 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:47.916 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.916 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.917 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.954 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:47.955 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:47.956 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [6ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [7ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [30ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [30ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@393441439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@13b6822e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b14ce23 -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489))) after lookup
18:50:47.958 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:47.962 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.962 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.963 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:47.993 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:47.995 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299998, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:47.998 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=110), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [6ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [7ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [30ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [30ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [70ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [75ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [76ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [104ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@393441439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@13b6822e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b14ce23 -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115))) after lookup
18:50:48.001 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:48.004 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.004 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.005 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.121 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:48.122 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:48.122 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=235), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [6ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [7ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [30ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [30ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [70ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [75ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [76ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [104ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [113ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [118ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [119ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [231ms] refreshing cache from master, [231ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [234ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@393441439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@13b6822e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115) -> (continuation of Deferred@1317578809 after retry RPC@2107268911), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b14ce23 -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115) -> (continuation of Deferred@1317578809 after retry RPC@2107268911))) after lookup
18:50:48.123 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:48.124 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.124 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.124 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.381 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:48.382 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129)[L],857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625),fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:48.382 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=495), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [6ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [7ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [30ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [30ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [70ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [75ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [76ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [104ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [113ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [118ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [119ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [231ms] refreshing cache from master, [231ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [234ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [235ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [237ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [237ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [491ms] refreshing cache from master, [491ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [494ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@393441439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@13b6822e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115) -> (continuation of Deferred@1317578809 after retry RPC@2107268911) -> (continuation of Deferred@1983724945 after retry RPC@1500604254), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b14ce23 -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115) -> (continuation of Deferred@1317578809 after retry RPC@2107268911) -> (continuation of Deferred@1983724945 after retry RPC@1500604254))) after lookup
18:50:48.383 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] connecting to peer
18:50:48.384 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.384 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] cleaning up while in state CONNECTING due to: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:48.384 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 32aab08028a947089da244a350b00c7e(127.3.97.65:42129) for tablet c92f0a97947e4a8e8362a7d223996d8b: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129
18:50:49.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.128189  6248 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
18:50:49.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.128546  6248 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:49.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.130546  6248 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 857be9840834437fb311c99b57bf0b4b (127.3.97.66:42625), 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:49.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:49.132763  6098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:49.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:49.134811  6098 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:49.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.139861  5967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "fa44f6302c4b4e11adf404ce13329e9d" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "857be9840834437fb311c99b57bf0b4b" is_pre_election: true
18:50:49.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.141047  6097 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e, 857be9840834437fb311c99b57bf0b4b
18:50:49.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.141578  6248 raft_consensus.cc:2726] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
18:50:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.295905  6253 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.296185  6253 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:49.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.296985  6253 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:49.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:49.300384  5884 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:49.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.305459  6176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d" is_pre_election: true
18:50:49.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.305902  6176 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 10.
18:50:49.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.307096  5884 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:49.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.307453  6253 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 11 FOLLOWER]: Leader pre-election won for term 12
18:50:49.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.307626  6253 raft_consensus.cc:482] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 32aab08028a947089da244a350b00c7e)
18:50:49.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.307744  6253 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 11 FOLLOWER]: Advancing to term 12
18:50:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.309827  6253 raft_consensus.cc:504] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.310402  6253 leader_election.cc:288] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 12 election: Requested vote from peers 32aab08028a947089da244a350b00c7e (127.3.97.65:42129), fa44f6302c4b4e11adf404ce13329e9d (127.3.97.67:39649)
18:50:49.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.311390  6176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c92f0a97947e4a8e8362a7d223996d8b" candidate_uuid: "857be9840834437fb311c99b57bf0b4b" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "fa44f6302c4b4e11adf404ce13329e9d"
18:50:49.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.311656  6176 raft_consensus.cc:3037] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 10 FOLLOWER]: Advancing to term 12
18:50:49.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:49.311946  5884 leader_election.cc:334] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111)
18:50:49.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.315984  6176 raft_consensus.cc:2454] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 857be9840834437fb311c99b57bf0b4b in term 12.
18:50:49.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.316601  5884 leader_election.cc:302] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 857be9840834437fb311c99b57bf0b4b, fa44f6302c4b4e11adf404ce13329e9d; no voters: 32aab08028a947089da244a350b00c7e
18:50:49.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.316946  6253 raft_consensus.cc:2781] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 12 FOLLOWER]: Leader election won for term 12
18:50:49.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.317512  6253 raft_consensus.cc:686] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [term 12 LEADER]: Becoming Leader. State: Replica: 857be9840834437fb311c99b57bf0b4b, State: Running, Role: LEADER
18:50:49.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.318176  6253 consensus_queue.cc:237] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:49.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.322440  3675 catalog_manager.cc:5495] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b reported cstate change: term changed from 11 to 12, leader changed from 32aab08028a947089da244a350b00c7e (127.3.97.65) to 857be9840834437fb311c99b57bf0b4b (127.3.97.66). New cstate: current_term: 12 leader_uuid: "857be9840834437fb311c99b57bf0b4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
18:50:49.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.901674  6176 raft_consensus.cc:1261] T c92f0a97947e4a8e8362a7d223996d8b P fa44f6302c4b4e11adf404ce13329e9d [term 12 FOLLOWER]: Refusing update from remote peer 857be9840834437fb311c99b57bf0b4b: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
18:50:49.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:49.902530  6275 consensus_queue.cc:1035] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
18:50:49.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:49.911265  5884 consensus_peers.cc:487] T c92f0a97947e4a8e8362a7d223996d8b P 857be9840834437fb311c99b57bf0b4b -> Peer 32aab08028a947089da244a350b00c7e (127.3.97.65:42129): Couldn't send request to peer 32aab08028a947089da244a350b00c7e. Status: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
18:50:50.141 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet c92f0a97947e4a8e8362a7d223996d8b for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865' with partition [<start>, <end>)
18:50:50.142 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=c92f0a97947e4a8e8362a7d223996d8b@[32aab08028a947089da244a350b00c7e(127.3.97.65:42129),857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)[L],fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)]}]
18:50:50.142 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=c92f0a97947e4a8e8362a7d223996d8b, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2255), Traces: [0ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [6ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [7ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [20ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [24ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [30ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [30ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [52ms] refreshing cache from master, [52ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [65ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [70ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [75ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [76ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [104ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [113ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [118ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [119ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [231ms] refreshing cache from master, [231ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [234ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [235ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [237ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [237ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [491ms] refreshing cache from master, [491ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [494ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, [496ms] sending RPC to server 32aab08028a947089da244a350b00c7e, [497ms] received response from server 32aab08028a947089da244a350b00c7e: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [497ms] delaying RPC due to: Network error: Failed to connect to peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129): Connection refused: /127.3.97.65:42129, [2251ms] refreshing cache from master, [2251ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.97.124:42301, [2254ms] Sub RPC GetTableLocations: received response from server master-127.3.97.124:42301: OK, deferred=Deferred@393441439(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$83/592786503@13b6822e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115) -> (continuation of Deferred@1317578809 after retry RPC@2107268911) -> (continuation of Deferred@1983724945 after retry RPC@1500604254) -> (continuation of Deferred@712862158 after retry RPC@213229774), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b14ce23 -> wakeup thread Time-limited test -> (continuation of Deferred@1648042015 after retry RPC@1150976489) -> (continuation of Deferred@926650980 after retry RPC@658430115) -> (continuation of Deferred@1317578809 after retry RPC@2107268911) -> (continuation of Deferred@1983724945 after retry RPC@1500604254) -> (continuation of Deferred@712862158 after retry RPC@213229774))) after lookup
18:50:50.143 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] connecting to peer
18:50:50.144 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] Successfully connected to peer
18:50:50.161 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xb6493dc2, L:/127.0.0.1:44622 - R:/127.3.97.66:42625] using TOKEN/null
18:50:50.185 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.3.97.65:42129
18:50:50.185 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.186278  3461 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task6_Crsr/build/asan/bin/kudu
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:50.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:50.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
18:50:50.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.440825  5884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.97.65:42129: connect: Connection refused (error 111) [suppressed 3 similar messages]
18:50:50.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.560504  6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
18:50:50.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.560814  6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
18:50:50.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.560899  6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
18:50:50.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.568576  6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
18:50:50.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.568774  6296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.97.65
18:50:50.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.578444  6296 tablet_server_runner.cc:78] Tablet server non-default flags:
18:50:50.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
18:50:50.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.3.97.84:44047
18:50:50.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.3.97.65:42129
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.3.97.65
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38971
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.3.97.126:46829,127.3.97.125:45183,127.3.97.124:42301
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.3.97.65
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/logs
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 11 Aug 2026 16:09:54 UTC on ccycloud.quasar-hitlqc.root.comops.site
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4580676
18:50:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
18:50:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.579908  6296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
18:50:50.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.581797  6296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
18:50:50.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.595819  6304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:50.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.595865  6307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:50.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 18:50:50.595778  6305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
18:50:50.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.596760  6296 server_base.cc:1034] running on GCE node
18:50:50.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.597491  6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
18:50:50.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.598492  6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
18:50:50.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.599756  6296 hybrid_clock.cc:648] HybridClock initialized: now 1786474250599683 us; error 44 us; skew 500 ppm
18:50:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.600121  6296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
18:50:50.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.602543  6296 webserver.cc:460] Webserver started at http://127.3.97.65:38971/ using document root <none> and password file <none>
18:50:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.603345  6296 fs_manager.cc:359] Metadata directory not provided
18:50:50.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.603509  6296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
18:50:50.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.608952  6296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
18:50:50.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.612228  6314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
18:50:50.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.613623  6296 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.613832  6296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data,/tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "32aab08028a947089da244a350b00c7e"
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 18:50:10 on dist-test-slave-pgkr"
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.615007  6296 fs_report.cc:383] FS layout report
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/wal
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/data
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
18:50:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
18:50:50.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.643898  6296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
18:50:50.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.644951  6296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
18:50:50.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.645314  6296 kserver.cc:163] Server-wide thread pool size limit: 3276
18:50:50.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.646564  6296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
18:50:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.648797  6322 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
18:50:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.654975  6296 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
18:50:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.655130  6296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
18:50:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.655294  6296 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
18:50:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.658522  6296 ts_tablet_manager.cc:610] Registered 1 tablets
18:50:50.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.658675  6296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
18:50:50.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.659085  6322 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap starting.
18:50:50.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.687876  6322 log.cc:824] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Log is configured to *not* fsync() on all Append() calls
18:50:50.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.712148  6296 rpc_server.cc:307] RPC server started. Bound to: 127.3.97.65:42129
18:50:50.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.712199  6448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.97.65:42129 every 8 connection(s)
18:50:50.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.713640  6296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6_Crsr/test-tmp/mini-kudu-cluster1973743100423989933/ts-0/data/info.pb
18:50:50.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.721035  3461 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6_Crsr/build/asan/bin/kudu as pid 6296
18:50:50.721 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
18:50:50.745 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:50.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.747408  6322 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
18:50:50.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.748221  6322 tablet_bootstrap.cc:492] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Bootstrap complete.
18:50:50.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.749960  6322 ts_tablet_manager.cc:1397] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent bootstrapping tablet: real 0.091s	user 0.068s	sys 0.013s
18:50:50.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.750689  6450 heartbeater.cc:347] Connected to a master server at 127.3.97.126:46829
18:50:50.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.751053  6450 heartbeater.cc:464] Registering TS with master...
18:50:50.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.751444  6449 heartbeater.cc:347] Connected to a master server at 127.3.97.124:42301
18:50:50.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.751598  6449 heartbeater.cc:464] Registering TS with master...
18:50:50.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.752230  6449 heartbeater.cc:511] Master 127.3.97.124:42301 requested a full tablet report, sending...
18:50:50.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.752274  6450 heartbeater.cc:511] Master 127.3.97.126:46829 requested a full tablet report, sending...
18:50:50.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.751053  6451 heartbeater.cc:347] Connected to a master server at 127.3.97.125:45183
18:50:50.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.754076  6451 heartbeater.cc:464] Registering TS with master...
18:50:50.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.754530  6451 heartbeater.cc:511] Master 127.3.97.125:45183 requested a full tablet report, sending...
18:50:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.755584  3509 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.755688  3675 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.755935  3592 ts_manager.cc:194] Re-registered known tserver with Master: 32aab08028a947089da244a350b00c7e (127.3.97.65:42129)
18:50:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.757174  3675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.97.65:58337
18:50:50.759 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1374) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786474211865, tablet=c92f0a97947e4a8e8362a7d223996d8b, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
18:50:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.759696  6322 raft_consensus.cc:348] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:50.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.760635  6322 raft_consensus.cc:729] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aab08028a947089da244a350b00c7e, State: Initialized, Role: FOLLOWER
18:50:50.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.761397  6322 consensus_queue.cc:260] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857be9840834437fb311c99b57bf0b4b" member_type: VOTER last_known_addr { host: "127.3.97.66" port: 42625 } } peers { permanent_uuid: "32aab08028a947089da244a350b00c7e" member_type: VOTER last_known_addr { host: "127.3.97.65" port: 42129 } } peers { permanent_uuid: "fa44f6302c4b4e11adf404ce13329e9d" member_type: VOTER last_known_addr { host: "127.3.97.67" port: 39649 } }
18:50:50.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.762804  6449 heartbeater.cc:503] Master 127.3.97.124:42301 was elected leader, sending a full tablet report...
18:50:50.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.765548  6322 ts_tablet_manager.cc:1428] T c92f0a97947e4a8e8362a7d223996d8b P 32aab08028a947089da244a350b00c7e: Time spent starting tablet: real 0.015s	user 0.011s	sys 0.004s
18:50:50.768 [DEBUG - main] (Connection.java:575) [peer 32aab08028a947089da244a350b00c7e(127.3.97.65:42129)] disconnecting while in state TERMINATED
18:50:50.770 [DEBUG - main] (Connection.java:575) [peer fa44f6302c4b4e11adf404ce13329e9d(127.3.97.67:39649)] disconnecting while in state TERMINATED
18:50:50.770 [DEBUG - main] (Connection.java:575) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] disconnecting while in state READY
18:50:50.770 [DEBUG - main] (Connection.java:575) [peer master-127.3.97.125:45183(127.3.97.125:45183)] disconnecting while in state READY
18:50:50.770 [DEBUG - main] (Connection.java:575) [peer master-127.3.97.124:42301(127.3.97.124:42301)] disconnecting while in state READY
18:50:50.770 [DEBUG - main] (Connection.java:575) [peer master-127.3.97.126:46829(127.3.97.126:46829)] disconnecting while in state READY
18:50:50.774 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.3.97.124:42301(127.3.97.124:42301)] handling channelInactive
18:50:50.774 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.3.97.124:42301(127.3.97.124:42301)] cleaning up while in state READY due to: connection closed
18:50:50.775 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.3.97.126:46829(127.3.97.126:46829)] handling channelInactive
18:50:50.775 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
18:50:50.775 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.3.97.126:46829(127.3.97.126:46829)] cleaning up while in state READY due to: connection closed
18:50:50.776 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] handling channelInactive
18:50:50.776 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 857be9840834437fb311c99b57bf0b4b(127.3.97.66:42625)] cleaning up while in state READY due to: connection closed
18:50:50.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.776597  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 6296
18:50:50.783 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.3.97.125:45183(127.3.97.125:45183)] handling channelInactive
18:50:50.783 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.3.97.125:45183(127.3.97.125:45183)] cleaning up while in state READY due to: connection closed
18:50:50.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.795486  6444 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:50.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.924543  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 6296
18:50:50.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.949659  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 5867
18:50:50.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:50.970221  6013 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:51.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.091432  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 5867
18:50:51.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.118211  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 6080
18:50:51.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.125789  6223 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.222586  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 6080
18:50:51.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.254109  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 3475
18:50:51.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.255678  3546 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:51.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.385007  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 3475
18:50:51.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.416764  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 3556
18:50:51.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.417955  3627 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:51.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.537557  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 3556
18:50:51.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.568270  3461 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6_Crsr/build/asan/bin/kudu pid 3636
18:50:51.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.569506  3709 generic_service.cc:196] Checking for leaks (request via RPC)
18:50:51.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 18:50:51.693962  3461 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6_Crsr/build/asan/bin/kudu with pid 3636
18:50:51.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T18:50:51Z chronyd exiting

Time: 45.653

OK (1 test)

